I've seen this before in the true random number generators I've been working on. Look at the following test. I've ent'd two jpegs, one 10 times the size of the other. A jpg is highly compressed so can't be compressed much further, and incompressibility is one of the definitions of random. It has however, some non random bytes such as control structures and the golden bytes. So it's almost random, as is your data.
C:\scratch>ent lada.jpg
Entropy = 7.956933 bits per byte.
Optimum compression would reduce the size
of this 4685491 byte file by 0 percent.
Chi square distribution for 4685491 samples is 319354.77, and randomly
would exceed this value less than 0.01 percent of the times.
Arithmetic mean value of data bytes is 132.0827 (127.5 = random).
Monte Carlo value for Pi is 3.027673947 (error 3.63 percent).
Serial correlation coefficient is 0.000597 (totally uncorrelated = 0.0).
C:\scratch>ent lena.jpg
Entropy = 7.947390 bits per byte.
Optimum compression would reduce the size
of this 408932 byte file by 0 percent.
Chi square distribution for 408932 samples is 33552.83, and randomly
would exceed this value less than 0.01 percent of the times.
Arithmetic mean value of data bytes is 116.3320 (127.5 = random).
Monte Carlo value for Pi is 3.357464603 (error 6.87 percent).
Serial correlation coefficient is 0.020913 (totally uncorrelated = 0.0).
Compressibility stays similarish as a jpeg is a jpeg. pi improves for the larger lada file somewhat as you have more samples that are randomish. However, Chi increased exponentially as you have more and more control structures (read non random values) and as Chi uses a squared function in the calculation this error builds rapidly.
In summary, there are some values in your data set that occur at way too wrong a frequency. A histogram of the data over the full 0-255 range in a spreadsheet /gnuplot might help to show something obvious, but with chi of only 400 it might be subtle. You're hampering your analysis with the small sample size. ent is typically run against an 11 MB data set to get a target Chi of about 250.
One methodology I find useful is to draw a picture of the data. Load your data file into a pixel editor like GIMP as raw data and create a square image. Subtle patterns will show up if you sit and stare at it for a few minutes.
There are two questions you must consider. One, have you cocked up your randomness extraction algorithm and it's something you can eventually fix? Two more worryingly, is are you totally, utterly and willing to stake your puppy's life on it convinced that the original source of your numbers is really random?
Supplemental:
I don't see dips in your histogram. I see waves. The black line identifies at least to me, two halves of the histogram. Eight less strong ones (pink line), then 16 (blue line) and so on in powers of 2. That kinda corresponds to bit positions. It seems fairly regular. Does that mean anything to you? I think that your seeing a strobe effect between threads and your timer /counter with some jitter overlaid. I think that the strobe effect is probably totally uniform. I think the jitter is the source of your randomness, and I think that it's true randomness. Congratulations.
If you're interested in pursuing the random generator thing, ent is naff at measuring entropy. Take the file you generated for the histogram, and compress it as hard as you can. I suggest the program fp8, but zipping will be a good start. The resultant file will be a very good measure of how many truly random bytes you're generated.
See, printing out an image (in whatever form) is useful.
(Help Centre will be angry with us for having this discussion).
