Tokenization
AIKernel.Dtos reference members from AIKernel.NET/src/AIKernel.Dtos/Tokenization.
Source: AIKernel.NET/src/AIKernel.Dtos/Tokenization
Summary
This section documents 20 generated members for AIKernel.Dtos. Member kinds in this source section: method, property, type.
Why
This reference page is intentionally grouped with package and source context so it is useful as documentation, not only as an autogenerated member list.
Usage
Use this page when you already know the package and need to inspect the source-backed member list from AIKernel.NET/src/AIKernel.Dtos/Tokenization.
Examples
dotnet add package AIKernel.Dtos --version 0.1.2Notes
Member descriptions come from XML documentation files when available. Empty cells mean the source does not currently provide XML documentation for that member in this language.
See Also
Package | Docs | Generation Report
| Kind | Member | Summary | Details |
|---|---|---|---|
| method | AIKernel.Dtos.Tokenization.TokenizerStatistics.Empty<string>M:AIKernel.Dtos.Tokenization.TokenizerStatistics.Empty<string> |
Executes the Empty<string> operation as part of the AIKernel public reference surface. | No additional XML detail. |
| property | AIKernel.Dtos.Tokenization.PaddingInfo.LogicalTokenCountP:AIKernel.Dtos.Tokenization.PaddingInfo.LogicalTokenCount |
Gets or sets the LogicalTokenCount value exposed by the AIKernel public reference surface. | No additional XML detail. |
| property | AIKernel.Dtos.Tokenization.PaddingInfo.MemoryDifferenceBytesP:AIKernel.Dtos.Tokenization.PaddingInfo.MemoryDifferenceBytes |
Gets or sets the MemoryDifferenceBytes value exposed by the AIKernel public reference surface. | No additional XML detail. |
| property | AIKernel.Dtos.Tokenization.PaddingInfo.PaddingAmountP:AIKernel.Dtos.Tokenization.PaddingInfo.PaddingAmount |
Gets or sets the PaddingAmount value exposed by the AIKernel public reference surface. | No additional XML detail. |
| property | AIKernel.Dtos.Tokenization.PaddingInfo.PaddingMethodP:AIKernel.Dtos.Tokenization.PaddingInfo.PaddingMethod |
Gets or sets the PaddingMethod value exposed by the AIKernel public reference surface. | No additional XML detail. |
| property | AIKernel.Dtos.Tokenization.PaddingInfo.PaddingPercentageP:AIKernel.Dtos.Tokenization.PaddingInfo.PaddingPercentage |
Gets or sets the PaddingPercentage value exposed by the AIKernel public reference surface. | No additional XML detail. |
| property | AIKernel.Dtos.Tokenization.PaddingInfo.PhysicalCardinalityP:AIKernel.Dtos.Tokenization.PaddingInfo.PhysicalCardinality |
Gets or sets the PhysicalCardinality value exposed by the AIKernel public reference surface. | No additional XML detail. |
| property | AIKernel.Dtos.Tokenization.PaddingInfo.RationaleP:AIKernel.Dtos.Tokenization.PaddingInfo.Rationale |
Gets or sets the Rationale value exposed by the AIKernel public reference surface. | No additional XML detail. |
| property | AIKernel.Dtos.Tokenization.Token.EndPositionP:AIKernel.Dtos.Tokenization.Token.EndPosition |
Gets or sets the EndPosition value exposed by the AIKernel public reference surface. | No additional XML detail. |
| property | AIKernel.Dtos.Tokenization.Token.StartPositionP:AIKernel.Dtos.Tokenization.Token.StartPosition |
Gets or sets the StartPosition value exposed by the AIKernel public reference surface. | No additional XML detail. |
| property | AIKernel.Dtos.Tokenization.Token.TokenIdP:AIKernel.Dtos.Tokenization.Token.TokenId |
Gets or sets the TokenId value exposed by the AIKernel public reference surface. | No additional XML detail. |
| property | AIKernel.Dtos.Tokenization.Token.TokenTypeP:AIKernel.Dtos.Tokenization.Token.TokenType |
Gets or sets the TokenType value exposed by the AIKernel public reference surface. | No additional XML detail. |
| property | AIKernel.Dtos.Tokenization.Token.ValueP:AIKernel.Dtos.Tokenization.Token.Value |
Gets or sets the Value value exposed by the AIKernel public reference surface. | No additional XML detail. |
| property | AIKernel.Dtos.Tokenization.TokenizerStatistics.AverageTokenLengthP:AIKernel.Dtos.Tokenization.TokenizerStatistics.AverageTokenLength |
Gets or sets the AverageTokenLength value exposed by the AIKernel public reference surface. | No additional XML detail. |
| property | AIKernel.Dtos.Tokenization.TokenizerStatistics.MaxTokenLengthP:AIKernel.Dtos.Tokenization.TokenizerStatistics.MaxTokenLength |
Gets or sets the MaxTokenLength value exposed by the AIKernel public reference surface. | No additional XML detail. |
| property | AIKernel.Dtos.Tokenization.TokenizerStatistics.VersionP:AIKernel.Dtos.Tokenization.TokenizerStatistics.Version |
Gets or sets the Version value exposed by the AIKernel public reference surface. | No additional XML detail. |
| property | AIKernel.Dtos.Tokenization.TokenizerStatistics.VocabularySizeP:AIKernel.Dtos.Tokenization.TokenizerStatistics.VocabularySize |
Gets or sets the VocabularySize value exposed by the AIKernel public reference surface. | No additional XML detail. |
| type | AIKernel.Dtos.Tokenization.PaddingInfoT:AIKernel.Dtos.Tokenization.PaddingInfo |
Represents the PaddingInfo public reference surface for AIKernel package consumers. | No additional XML detail. |
| type | AIKernel.Dtos.Tokenization.TokenT:AIKernel.Dtos.Tokenization.Token |
Represents the Token public reference surface for AIKernel package consumers. | No additional XML detail. |
| type | AIKernel.Dtos.Tokenization.TokenizerStatisticsT:AIKernel.Dtos.Tokenization.TokenizerStatistics |
Represents the TokenizerStatistics public reference surface for AIKernel package consumers. | No additional XML detail. |