Take the 2-minute tour ×
Cryptography Stack Exchange is a question and answer site for software developers, mathematicians and others interested in cryptography. It's 100% free, no registration required.

Are there online resources with an ASCII character frequency table for HTML?

There's tons of them for english, but I can't seem to find one for page source. I know that this is something that's likely to vary from site to site, but surely there's rule-of-thumb good-enough-usually tables where < is ranked higher than z?

share|improve this question
    
HTML has lots of repetitive strings (e.g. tag names), so you may want to try looking at $n$-gram frequencies for $n\ge2$, or just crib dragging. –  Ilmari Karonen Jun 22 at 18:15

3 Answers 3

Here's a simple PHP script that calculates bytes frequencies from 10 random Wikipedia articles:

<?php

$frequencies = array();
    $total = 0;
for ($i=0; $i<256; $i++) $frequencies[$i] = 0;
    for ($i=0; $i<10; $i++) {
  $src = file_get_contents('https://en.wikipedia.org/wiki/Special:Random');
      foreach (str_split($src) as $char) {
        $frequencies[ord($char)]++;
        $total++;
      }
    }
    header('Content-Type: text/plain');
    for ($i=0; $i<256; $i++) {
  printf('%3d:%7.4f%s',$i,$frequencies[$i]*100/$total,(($i&7)==7)?"\n":'  ');
}

You could easily extend this to fetch more pages from a wider range of sources.

Here are the results I got:

  0: 0.0000    1: 0.0000    2: 0.0000    3: 0.0000    4: 0.0000    5: 0.0000    6: 0.0000    7: 0.0000
  8: 0.0000    9: 1.9908   10: 1.0258   11: 0.0000   12: 0.0000   13: 0.0000   14: 0.0000   15: 0.0000
 16: 0.0000   17: 0.0000   18: 0.0000   19: 0.0000   20: 0.0000   21: 0.0000   22: 0.0000   23: 0.0000
 24: 0.0000   25: 0.0000   26: 0.0000   27: 0.0000   28: 0.0000   29: 0.0000   30: 0.0000   31: 0.0000
 32: 4.8680   33: 0.0277   34: 4.5121   35: 0.0942   36: 0.0042   37: 0.2563   38: 0.2139   39: 0.0867
 40: 0.1498   41: 0.1498   42: 0.0275   43: 0.1042   44: 0.4450   45: 1.1506   46: 1.1177   47: 3.1013
 48: 0.6391   49: 0.4991   50: 0.5280   51: 0.3287   52: 0.2134   53: 0.2927   54: 0.2862   55: 0.1731
 56: 0.2161   57: 0.2103   58: 0.6376   59: 0.3994   60: 2.8618   61: 2.1683   62: 2.8618   63: 0.0682
 64: 0.0000   65: 0.3892   66: 0.1531   67: 0.3726   68: 0.1941   69: 0.1862   70: 0.2055   71: 0.0938
 72: 0.0913   73: 0.1354   74: 0.0572   75: 0.1103   76: 0.1240   77: 0.1656   78: 0.1242   79: 0.0963
 80: 0.2883   81: 0.0468   82: 0.1858   83: 0.3152   84: 0.2528   85: 0.0832   86: 0.0666   87: 0.1076
 88: 0.0060   89: 0.0354   90: 0.0237   91: 0.0807   92: 0.0522   93: 0.0807   94: 0.0079   95: 0.8702
 96: 0.0000   97: 6.3074   98: 0.9143   99: 2.2221  100: 2.8104  101: 6.4384  102: 1.2492  103: 1.6232
104: 1.9022  105: 6.5891  106: 0.0815  107: 1.2405  108: 3.9219  109: 1.5149  110: 3.4597  111: 3.1206
112: 2.1826  113: 0.0285  114: 3.6109  115: 3.5510  116: 5.1196  117: 1.3010  118: 0.7861  119: 1.3322
120: 0.5001  121: 0.7843  122: 0.1313  123: 0.0483  124: 0.0000  125: 0.0483  126: 0.0000  127: 0.0000
128: 0.0431  129: 0.0044  130: 0.0079  131: 0.0152  132: 0.0017  133: 0.0029  134: 0.0002  135: 0.0033
136: 0.0035  137: 0.0015  138: 0.0023  139: 0.0017  140: 0.0031  141: 0.0012  142: 0.0006  143: 0.0006
144: 0.0033  145: 0.0012  146: 0.0017  147: 0.0285  148: 0.0012  149: 0.0021  150: 0.0029  151: 0.0037
152: 0.0017  153: 0.0046  154: 0.0006  155: 0.0012  156: 0.0023  157: 0.0004  158: 0.0023  159: 0.0173
160: 0.0025  161: 0.0017  162: 0.0027  163: 0.0021  164: 0.0069  165: 0.0094  166: 0.0021  167: 0.0035
168: 0.0037  169: 0.0025  170: 0.0035  171: 0.0025  172: 0.0025  173: 0.0037  174: 0.0027  175: 0.0004
176: 0.0123  177: 0.0050  178: 0.0069  179: 0.0042  180: 0.0046  181: 0.0031  182: 0.0021  183: 0.0025
184: 0.0056  185: 0.0044  186: 0.0048  187: 0.0046  188: 0.0025  189: 0.0021  190: 0.0027  191: 0.0037
192: 0.0000  193: 0.0000  194: 0.0092  195: 0.0094  196: 0.0012  197: 0.0006  198: 0.0000  199: 0.0000
200: 0.0000  201: 0.0002  202: 0.0002  203: 0.0000  204: 0.0000  205: 0.0000  206: 0.0054  207: 0.0002
208: 0.0202  209: 0.0104  210: 0.0004  211: 0.0000  212: 0.0004  213: 0.0031  214: 0.0004  215: 0.0000
216: 0.0089  217: 0.0073  218: 0.0002  219: 0.0025  220: 0.0000  221: 0.0000  222: 0.0000  223: 0.0000
224: 0.0125  225: 0.0023  226: 0.0433  227: 0.0015  228: 0.0019  229: 0.0037  230: 0.0031  231: 0.0017
232: 0.0017  233: 0.0010  234: 0.0008  235: 0.0012  236: 0.0017  237: 0.0008  238: 0.0000  239: 0.0029
240: 0.0171  241: 0.0000  242: 0.0000  243: 0.0000  244: 0.0000  245: 0.0000  246: 0.0000  247: 0.0000
248: 0.0000  249: 0.0000  250: 0.0000  251: 0.0000  252: 0.0000  253: 0.0000  254: 0.0000  255: 0.0000

As you would expect, the frequencies for < (60) and > (62) are identical.

share|improve this answer
    
Nice! It's almost an entry to Programming Puzzles & Code Golf –  cygnusv Jun 22 at 16:33

I don't know of any reference to such information, but it's something you can easily estimate with a very simple program.

Just for illustration, I have made a very quick analysis of the HTML code of this very page using an frequency analysis tool (http://www.dcode.fr/frequency-analysis). These are the results (only the most frequent characters are shown):

enter image description here

Of course, this is just a simplified example of analysis, and for real-world purposes you would need a much bigger sample, but the process itself is very simple.

share|improve this answer
    
What are 2A and 15A? $\:$ (I would imagine that one of them is space and the other is tab or newline.) $\hspace{.54 in}$ –  Ricky Demer Jun 22 at 8:20
    
@RickyDemer I don't know, the online tool only prints a blank space when presenting the results. I guess that 2A is space, and 15A is newline. –  cygnusv Jun 22 at 8:29

There is another angle that can add some accuracy to what was suggested above which is to count in the frequency a page (or domain) is visited on the web when you do your frequency calculation. For example, a tag that google.com uses should have a high frequency value from the perspective of a proxy or firewall serving average users. Even though that tag might never show up anywhere else on the web (not google.com.)

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.