Abstract:
Similarly, to do all 10 programming tasks correctly, the new version of DeepSeek cost less than 40% of Claude Opus 5.5, and the average wait for each task was two minutes longer.
On October 17, DeepSeek released V4.1-Flash-1017, and the price remained unchanged. After its release, developers have already used it to make games, build web pages, and modify code.
In order to test its actual level, developer Ibragim Badertdinov selected 10 tasks from his daily work and ran different models three times in the same set of programming tools, using the high inference gear of DeepSeek V4.1-Flash-1017 and the low gear of Opus 5.5 and GPT-6 Astra.
After actual testing, all three models passed the test, but the gap was widened by billing and time-consuming: DeepSeek only spent an average of US$0.045 per task, which was far lower than Opus's US$0.1168 and GPT-6 Astra's approximately US$0.26; however, Opus and GPT only took 37 seconds and 55 seconds to complete, respectively, but DeepSeek took 2 minutes and 38 seconds.
These 10 tasks are all small in scale. As a complete project, is the gap between DeepSeek, GPT, and Claude just a matter of waiting a little longer?
Passed all ten questions in 01, DeepSeek saves money, Opus saves time
The topics Ibragim picked came from about three months of his own records of using agents, involving data processing, back-end, front-end and deployment, using Python and TypeScript. He picked out the tasks that he could usually complete in less than an hour, and then added automatic checking programs to them.
A simplified example he disclosed is to import a batch of data into a local data table. In addition to successfully importing, the program must also skip duplicate records, reject conflicting numbers, and report unsatisfactory data; when replacing old data, it cannot destroy the original content halfway through writing. The test will check for empty input, data conflicts, original file corruption, etc.
All three models passed these checks, but DeepSeek took more steps.
Also running on the Pi programming tool, DeepSeek takes an average of 15.3 steps per task, Opus takes 3.7 steps, and GPT-6 Astra takes 6.5 steps. The total token usage of DeepSeek is about 547,000, and that of Opus is about 31,000.I use a lot, but the bill doesn't go up. Ibragim explained that almost all DeepSeek inputs hit the cache, and the recorded cache hit rate reached 97.9%. The 547,000 tokens were not newly written by the model. Such a long answer included content that was repeatedly sent in multiple rounds of calls. Cheap cached inputs kept the cost of this test under 5 cents per test.
For users, whether these two minutes have any impact depends on whether they have been waiting in front of the screen. After handing over the task and then doing other things, waiting may not be a hindrance; if you want to make the next round of modifications immediately after seeing the results, the speed of Opus and GPT in this set of tests is more useful. What is compared here is the cost of completing tasks through the API, not the monthly fee for ChatGPT or Claude subscription packages.
The new version is also improved compared to the previous version. Ibragim previously used the highest inference setting of the September version of Flash to test, and passed 28 times out of 30 runs, with an average time of 4 minutes and 30 seconds for each test. This time, using the new version of the high-end setting, it passed all 30 times, and the time was reduced to 2 minutes and 38 seconds.
02 I built a BMW for $0.38, but it took 40 minutes to run an ordinary web page
What DeepSeek wants to show this time is not just to fill in a function. Officially released cases include the Forbidden City 3D scene, a web racing game, and a "Portal 2" reproduction demonstration produced with Godot. The company says the new version improves the ability to complete projects from scratch through reinforcement learning with self-feedback.
Developer Fabiano Firmo asked him to use Three.js to create a 3D model of the BMW M5. In the picture he posted, a red car appears on the display page, with color options next to it.
According to him, he only gave one reminder this time, which cost $0.38 and took 1 hour and 10 minutes.
Some users also demonstrated a game similar to "Rocket League", where cars play football on the field. X user Markets & Mayhem commented that the physics effects work normally and it is playable, but it still needs polishing. For someone who wants to prototype a web page or game first, these demos are easier to see the use for than model exam scores.
However, being willing to wait more than an hour to see a 3D car does not mean that you are willing to wait that long to create an ordinary web page.
Developer Alex Carter said that he asked DeepSeek Harness to create an ordinary landing page, which took 41 minutes to run; the next two web page tasks took another 43 minutes and 40 minutes respectively. He thought the model was smart, cheap, and presented well, but the pages were years of work and could usually be completed in ten minutes.
Developer unbug also has a similar feeling: the model has become smarter, but it is easy to think too long and delay taking action. Another user Elshayib emphasized that his complaint was not about the number of words output per second, but that the model took too long to think before acting.
A test of the API platform OmniaKey recorded this long reasoning output. It requires the new version of Flash to write a program that calculates expressions according to Python operation rules, and cannot call ready-made parsing tools. The model took about 88 seconds to pass all 30 inspection use cases; of the approximately 20,700 tokens output, approximately 19,700 were considered content.
But Flash is not slow for all tasks. In another test of OmniaKey, it extracted 10 fields from an invoice email that was mixed in German and English, and answered all of them correctly in 5.2 seconds; the tool call to query the weather and time in two cities was completed in 2.6 seconds. OmniaKey itself sells the model API, and each of this set of tests will only be run once.
These examples of BMW modeling and landing pages are not comparisons of the three models on the same topic. How deepSeek's finished product is and how many times it will be revised later still need to be compared with GPT and Claude under the same requirements.
03 After lowering the gear, it was one minute faster and there were three more failures
Since the high inference setting takes a long time, what will happen if it is lowered? Ibragim also tested it.
In Pi, after the new version of DeepSeek is adjusted to the low setting, the average time per item is shortened from 2 minutes and 38 seconds to 1 minute and 31 seconds, and the cost is reduced from US$0.045 to US$0.0257. However, 3 out of 30 runs failed.
All passes obtained at high levels cannot be retained intact.
At this low level, I switched to DeepSeek's own Harness, and the results changed again: it passed 29 times in 30 runs, two more than Pi. According to Ibragim's records, the official tool environment helped the model handle a boundary condition that was missed in Pi.
The two sets of tools work differently with models. DeepSeek Harness directly provides tools for reading and editing files, while Pi mainly allows the model to be operated through the command line; the system prompt words of the former are also longer. After changing tools this time, token usage increased by about 75%, and the average time spent increased from 1 minute and 31 seconds to 1 minute and 53 seconds. In both sets of tools, only 9 tasks can be passed three times in a row.
For the same model, if you change the inference gear and set of tools, the speed, cost and success will change.
You only see the results posted by others, but you also need to know what settings they used.DeepSeek also clearly mentioned when it was released that it had optimized tools such as DeepSeek Harness, Tencent CodeBuddy and WorkBuddy, and recommended Harness users to upgrade to v0.2.1-rc.1 or higher.
This update has been launched on web pages, apps and APIs. API calls still use deepseek-flash, and the price remains unchanged; the new model weights have also been made public under the MIT protocol.
However, the previous programming test was completed by connecting the model to a tool that can read files, modify code, and run checks. Asking it to write a piece of code in the chat window is different from asking it to take over a project. The usage environment is different.
04For the same set of questions, GPT and Claude can be cheaper
Opus and Astra are the two companies' flagships, and OpenAI and Anthropic also have cheaper models.
In the same set of Ibragim tests, GPT-6 Luna's xhigh gear and Haiku 5.5's high gear all passed 30 times. The average cost of each item was only US$0.0145 and US$0.0208 respectively, which is lower than DeepSeek.
The average time consumption of the two is 2 minutes and 46 seconds and 2 minutes and 08 seconds respectively.This makes the choice no longer just "expensive GPT, Claude" or "cheap DeepSeek". People who already use Luna or Haiku to handle small tasks may not be able to save further money by switching to Flash.
Harder tasks change the question from how long to wait to whether you can trust the results. X user Yum said that after trying the new version, he is still most worried about hallucinations, especially the reconstruction and architecture modification across multiple files. He would have another model check the changes rather than let DeepSeek approve his own solution. This is his choice of usage, not a test on the same question to prove that GPT or Claude is more reliable.
Therefore, the most solid progress at present is that the new version of DeepSeek has delivered the same inspection results as the flagship model in a batch of real small programming tasks.
How many manual corrections are needed in large projects and whether problems can be repaired will determine how far this replacement can go.
There is still a lot of work between building a 3D car and maintaining a project over the long term.Ibragim is also preparing to make this test more difficult. He said at the end of the post that the next step would be to select a group of much more difficult tasks and compare different tools and models.
Comments