How much trouble can a little emoticon cause? You won't think of it. This emoticon comes from the 20th episode of the ninth season of the classic American TV series "Friends" (also known as "Six"). Rachel, played by Jennifer Aniston, heard that a soap opera party was going to be held on the rooftop and she immediately cheered. It is only 1.6MB in size and has been used 246,173 times on a social platform called Discourse because it is so popular among netizens.
Every time it is used, it will be backed up repeatedly, eventually forming 377GB of redundant backup and more than 240,000 hard links, which directly breaks through the Linux file system capacity limit and causes the backup mechanism to fail.
As an open source software project, Discourse provides technical support for more than 22,000 online communities. The real-time chat function supports the insertion of emoticons and GIF animations.
However, it has a special "safe upload" mechanism:When a file is moved between different security scenarios, such as forwarding from a private message to a public post, the system generates a new copy with a random SHA1 encryption value. Although the file has not changed at all, Discourse will treat it as a new file.
Therefore, if a popular picture or emoticon is continuously spread in posts, forwards, and private messages, a new copy will be generated every time.
In fact, Discourse has long been aware of the problem of being overwhelmed by duplicate files. The initial solution was to track the original files through file Hash values. When backing up, uploaded files are grouped by hash values. Only the first file in each group is downloaded, and hard links are created for duplicate files.
It looks good and elegant, but the Discourse Linux system uses the most common and oldestThe ext4 file system supports a maximum of 16TB single file and 1EB file system, but each file only allows a maximum of 65,000 hard links.
Therefore, the Discourse solution cannot download more than 240,000 duplicate files only once. After reaching the upper limit, in addition to the first download, the system also performed approximately 181,000 additional backup downloads.
That is to say,What crashes the system is not the huge backup files, but the crazy number of hard links.
Fortunately,Discourse finally found the perfect solution. The idea was to create hard links. However, when the file system gave an EMLINK error message of "too many links", it copied a corresponding file locally and set the new file as the "master file". Based on it, it continued to create hard links until the link limit was reached again.
Discourse is quite satisfied that this new measure works on all file systems and requires no additional configuration.
This is perfectly acceptable as long as the file system cannot be changed.
Finally, Discourse also joked,Now we know that Jennifer Aniston can also test server stress.
