{"id":777,"date":"2024-05-26T11:43:10","date_gmt":"2024-05-26T03:43:10","guid":{"rendered":"http:\/\/madapapa.com\/wordpress\/?p=777"},"modified":"2024-05-26T22:33:36","modified_gmt":"2024-05-26T14:33:36","slug":"wen-ben-xiang-liang-hua-he-ci-pin-yi-ji-xi-shu-ju","status":"publish","type":"post","link":"http:\/\/madapapa.com\/wordpress\/?p=777","title":{"rendered":"\u6587\u672c\u5411\u91cf\u5316\u548c\u8bcd\u9891\uff0c\u4ee5\u53ca\u7a00\u758f\u77e9\u9635\u548c\u7a20\u5bc6\u77e9\u9635"},"content":{"rendered":"

<\/span><\/a>\u6587\u672c\u5411\u91cf\u5316\u548c\u8bcd\u9891\u7edf\u8ba1\u793a\u4f8b<\/h2>\n

\u4e0b\u9762\u7684\u4ee3\u7801\u628a\u4e00\u4e2a\u6587\u6863\uff0c\u8f6c\u5316\u6210\u7a00\u758f\u77e9\u9635\uff0c\u7136\u540e\u4e3a\u4e86\u66f4\u901a\u4fd7\u6613\u61c2\u7684\u5904\u7406\uff08\u540c\u65f6\u90e8\u5206\u7b97\u6cd5\u4e5f\u4e0d\u652f\u6301\u7a00\u758f\u77e9\u9635\uff09\uff0c\u518d\u628a\u5b83\u8f6c\u5316\u4e3a\u7a20\u5bc6\u77e9\u9635<\/p>\n

from sklearn.feature_extraction.text import CountVectorizer\n\n# Sample text data\ndocuments = [\n    "I love programming in Python",\n    "Python is a great language",\n    "I love coding"\n]\n\n# Create an instance of CountVectorizer\nvect = CountVectorizer()\n\n# Fit and transform the data\nX = vect.fit_transform(documents)\n\n# Convert to dense array\nX_dense = X.toarray()\n\n# Get feature names (tokens)\nfeature_names = vect.get_feature_names_out()\n\n# Print feature names and the dense array for verification\nprint("Feature names:", feature_names)\nprint("Dense array:\\n", X_dense)\n\n# Sum the counts of each token across all documents\ntoken_counts = X_dense.sum(axis=0)\n\n# Create a dictionary of tokens and their counts\ntoken_count_dict = dict(zip(feature_names, token_counts))\n\n# Print the token counts\nfor token, count in token_count_dict.items():\n    print(f"{token}: {count}")\n\n<\/code><\/pre>\n
<\/span><\/a>\u8f93\u51fa\u7684\u7ed3\u679c\u5982\u4e0b<\/h2>\n
This example will print the feature names, the dense array, and the token counts:<\/p>\n
Feature names: ['coding' 'great' 'in' 'is' 'language' 'love' 'programming' 'python']\nDense array:\n [[0 0 0 0 0 1 1 1]\n  [0 1 1 1 1 0 0 1]\n  [1 0 0 0 0 1 0 0]]\ncoding: 1\ngreat: 1\nin: 1\nis: 1\nlanguage: 1\nlove: 2\nprogramming: 1\npython: 2\n\n<\/code><\/pre>\n
<\/span><\/a>Explanation of the Output<\/h2>\n
Feature Names:<\/p>\n
The feature names are printed in the same order as they appear in the dense array’s columns: [‘coding’ ‘great’ ‘in’ ‘is’ ‘language’ ‘love’ ‘programming’ ‘python’].
\nDense Array:<\/p>\n
The dense array shows the token counts for each document, with each column corresponding to the respective feature name.
\nFor example, the first column corresponds to ‘coding’, the second column to ‘great’, and so on.
\nToken Counts:<\/p>\n
The token counts dictionary shows the total count of each token across all documents, matching the counts in the dense array.
\nVerification
\nTo verify the correspondence, look at the dense array and the feature names:<\/p>\n
The first column in X_dense corresponds to ‘coding’. In the dense array, the first column has counts [0, 0, 1], meaning ‘coding’ appears once in the third document.
\nThe second column corresponds to ‘great’. The counts are [0, 1, 0], meaning ‘great’ appears once in the second document.
\nThis pattern continues for all feature names and their corresponding columns.
\nConclusion
\nThe sequence of the feature names is the same as the columns of the dense array. Each column in the dense array represents the count of a specific token, and the order of these tokens is given by feature_names.<\/p>\n
<\/span><\/a>Sparse Matrix vs. Dense Array<\/h2>\n
When using CountVectorizer to transform text data into a matrix of token counts, the result is a sparse matrix by default. Let’s explore the differences between sparse matrices and dense arrays, and why one might be preferred over the other in certain contexts.<\/p>\n
<\/span><\/a>Sparse Matrix<\/h3>\n
A sparse matrix is a matrix in which most of the elements are zero. Instead of storing every element, sparse matrices store only the non-zero elements and their positions. This can lead to significant memory savings when dealing with large datasets where the number of zeros vastly outnumbers the number of non-zero elements.<\/p>\n
Advantages:
\nMemory Efficiency: Sparse matrices save memory by only storing non-zero elements. This is crucial for large datasets with many features (e.g., in text processing where there are many words but each document only contains a small subset).
\nPerformance: Certain operations can be faster on sparse matrices due to the reduced amount of data.
\nDisadvantages:
\nComplexity: Sparse matrices are more complex to manipulate and understand because they don’t store data in a straightforward row-by-row manner.<\/p>\n
<\/span><\/a>Dense Array<\/h3>\n
A dense array, on the other hand, stores all elements explicitly, including the zero elements. This means it takes up more memory but is simpler to understand and manipulate.<\/p>\n
Advantages:
\nSimplicity: Dense arrays are easier to work with because they store data in a straightforward manner, where each element corresponds directly to a position in the matrix.
\nCompatibility: Some algorithms and libraries work only with dense arrays, not sparse matrices.
\nDisadvantages:
\nMemory Usage: Dense arrays can consume a lot of memory if the dataset is large and contains many zero elements.<\/p>\n
<\/span><\/a>\u793a\u4f8b\u89e3\u91ca<\/h3>\n
Interpretation
\nSparse Matrix:<\/p>\n
Efficiently stores data when most elements are zero.
\nExample representation (only showing non-zero values and their positions):
\nscss<\/p>\n
(0, 4) 1\n(0, 7) 1\n(0, 8) 1\n(0, 5) 1\n(0, 6) 1\n(1, 4) 1\n(1, 2) 1\n(1, 1) 1\n(1, 3) 1\n(2, 5) 1\n(2, 0) 1\n(2, 6) 1\n<\/code><\/pre>\n
Dense Array:<\/p>\n
Simpler to understand as it stores all elements explicitly.
\nExample representation:<\/p>\n
[[0 0 0 1 1 0 1 1 1]\n [0 0 1 0 1 1 0 0 0]\n [1 0 0 0 0 1 1 0 0]]\n<\/code><\/pre>\n
Each row corresponds to a document, and each column corresponds to a token. The values represent the count of each token in the respective document.<\/p>\n
In summary, sparse matrices are memory-efficient and suitable for large datasets with many zero elements, while dense arrays are straightforward and easier to work with for smaller datasets or when simplicity is desired.<\/p>\n
<\/span><\/a>\u6765\u81eaKimi\u7684\u4e2d\u6587\u89e3\u91ca<\/h2>\n
\u5728\u8ba1\u7b97\u673a\u79d1\u5b66\u548c\u6570\u5b66\u4e2d\uff0c\u7a00\u758f\u77e9\u9635\u548c\u7a20\u5bc6\u77e9\u9635\u662f\u4e24\u79cd\u4e0d\u540c\u7c7b\u578b\u7684\u77e9\u9635\uff0c\u5b83\u4eec\u5728\u5b58\u50a8\u548c\u5904\u7406\u4e0a\u6709\u6240\u4e0d\u540c\uff0c\u4e3b\u8981\u53d6\u51b3\u4e8e\u77e9\u9635\u4e2d\u975e\u96f6\u5143\u7d20\u7684\u6570\u91cf\u3002<\/p>\n
<\/span><\/a>\u7a20\u5bc6\u77e9\u9635\uff08Dense Matrix\uff09<\/h3>\n
\u7a20\u5bc6\u77e9\u9635\u662f\u6307\u5927\u591a\u6570\u5143\u7d20\u90fd\u662f\u975e\u96f6\u7684\u77e9\u9635\u3002\u5728\u7a20\u5bc6\u77e9\u9635\u4e2d\uff0c\u975e\u96f6\u5143\u7d20\u7684\u6570\u91cf\u63a5\u8fd1\u4e8e\u77e9\u9635\u7684\u603b\u5143\u7d20\u6570\u91cf\u3002\u7a20\u5bc6\u77e9\u9635\u901a\u5e38\u4f7f\u7528\u5b8c\u6574\u7684\u4e8c\u7ef4\u6570\u7ec4\u6765\u8868\u793a\uff0c\u6bcf\u4e2a\u5143\u7d20\u90fd\u6709\u4e00\u4e2a\u5bf9\u5e94\u7684\u5b58\u50a8\u7a7a\u95f4\u3002<\/p>\n
\u7279\u70b9<\/strong>\uff1a<\/p>\n