Modx and limit of 5,000 documents

Introduction


Modx is a wonderful framework, but the resources and sections dedicated to modx, you can read the posts about some freimark limit of 5,000 documents and the customers happens asking will the site work if pages are more than 5 thousand.
You've probably already guessed, we will focus on modx evolution (version 1.0.5).

When there is a task to make the site more business cards, the question arises: how many pages can serve cmf/cms and how?

Modx is famous for its flexibility and for almost any task you can think of a few solutions, but the bottleneck — caching, specifically we are interested in the file assets/cache/siteCache.idx.php which contains everything that can be cached (except the pages that have your cache file assets/cache/docid_<ID>.pageCache.php).
To get around the slight inconvenience that may arise (if you make a portal and keep all the documents of modx) in the large site with the current concept of modx caching in several ways, which are a little lower.

What's wrong with caching


All of them so, but there is one moment — when the cache is cleared, the main cache file siteCache.idx.php should peresopnitsa again.

cache refresh provoke events such as any changes to the TV setting(s), chunks, snippets, templates, document changes (added a new resource, or removed from any other — you need to "refresh" the array of document IDs), as a result, if there is a website directory with several hundred papers in more than half of which dozens of TV options- size siteCache.idx.php can be several MB, and increase at the same time to form a new cache file you need to perform a dozen sql queries, if not more, the file is successfully overwritten as long as there is enough allocated memory (memory_limit) as the step of forming the main cache file, and other requests for the formation of the current page (where you currently are), otherwise will get an error: Fatal error: Allowed memory size of ... bytes exhausted (tried to allocate ... bytes) in ..assets/cache/siteCache.idx.php on line ....

Typical errors in large sites


Under the big website will be a resource in several thousand documents, whose number is constantly increasing.

At the wrong design of the website, the main burden will fall on the file system, specifically — on the record siteCache.idx.php consider one of the possible options.

for Example, a blog on modx

Suppose there is a blog on modx, the content of which "generate" users, each placing their articles, which are written in the database as a document and displayed in the modx document tree, that is quite logical and even convenient since you don't need to do this extra work, a separate table controllers to manage data on behalf of the user and the administrator (it's a few hours work, depending on the complexity), and tools to edit your site admin/Manager of fron-ends and there "out of the box" is quite convenient.

When adding a new document in modx, it is necessary to clear the entire cache, so at a frequency of 2-3 per minute document the site may slow down, with growth in the number of documents of the brakes increase with the time the website is not enough allocated memory and the server will return an error which begins: "Allow memory size of...".
You can increase the size of the allocated memory, but can not indefinitely increase (we will not say why), will, sooner or later, to reconsider the concept of our blog and most of the work to rewrite.

Testing modx


Not to be unfounded, have made a small test in the home, modelling the example of "a blog on modx".

test Objective

To check personally that the ehpm can work with more than 5,000 documents, to establish the conditions under which modx might stop working.
Tools

Laptop: CPU Core i3 2.13 Ghz, RAM 8Gb DDR3, HDD 500gb 5400об.min.
Software: Windows 7 x64, Apache 2.2.12, php 5.3.0, mysql 5.1.36
Server settings: conventional web server xampp, usual settings, the value of the memory_limit set to 64Mb.

Plan of work

    the
  1. to install modx evolution 1.0.5
    2. the
  2. to create a: 30 copies of template, 50 chunks, snippets 30, 100 TV parameters, 5,000 documents
    3. the
  3. to write a script which in part will make paragraph 2, perform paragraph 2.
    4. the
  4. if admin does not work — do item 6, if works — do item 5
    5. the
  5. increases the number of documents for 5,000, look item 4
    6. the
  6. increase memory_limit to 128 MB, if there are no errors — see item 5.
    7. the
  7. if you reached to this point, then got the error "Allowed memory size of...", concludes.


Progress

Set (start and finish) framework, along with all the chunks and snippets, which is enabled by default.

In the testing stage is not essential, what data will contain templates, chunks, snippets, TV settings, and documents. We will always copy one and the same element.
For copy operations resources have been written a little script:
    the
  1. source
    2. the
  2. copy to assets/snippets/test/test.php
    3. the
  3. start index-ajax.php?q=assets/snippets/test/test.php

In the admin made a copy of the first document (id = 1), as a child of the copied (id=2) set the document id=1, and forbidden to display the document in the menu (when there will be a lot of documents don't have to see them all on the site), the script will copy the created document (id=2) and all the clones will be as a parent to have a first document (id=1).

Copying resources



The initial size siteCache.idx.php — 161Кб.

Templates, the donor will be "MODxHost" — about the same amount of space occupied by any other templates in most projects, made 30 copies, also copied 100 TV parameters, 50 chunks (donor made WebLoginSideBar), 30 snippets (donor — AjaxSearch, a lot of the code contains what we need).

After these simple operations, the size siteCache.idx.php increased to rate 1,370 MB, website and admin panel is working, create a lot of documents (remember, the script duplicates the document with id=2, for id=2 we insert a little content, for example an article from somewhere, I copied here a piece).

test Results in table


image

Attempt to create another 5K documents with memory_limit=128Mb and the presence of 60K documents were unsuccessful.

When testing in the admin area will not load documents in the document tree, even though the admin panel was working, it is due to the fact, that copied all the documents to one "parent" (id=1), and to select and output the desired child document is not enough allocated memory.

When you create the cache file siteCache.idx.php it is worth considering what kind of page is created the main cache file, so as code snippets and plugins that are used on a particular page, is also depleting for PHP memory, because the maximum number of documents depends on how well-written code snippets and how to build queries.

The test is over, when memory_limit 64MB and 128MB of approximately 15K and 60K, respectively, can be maintained in modx, this is clearly more than the stated limits in the 5K documents, of course this is not the standard, but there is an indication that modx is, in principle, can work with a large number of documents.

Organization structure


To avoid the problem of lack of allocated memory, I can recommend the principles that should be followed when creating a website or portal on modx:
the
    the
  • do not write very many subsidiary documents for one parent to try to logically distribute documents to groups;
    • the
  • dynamic content for huge directories should be organized in a separate table, write it need snippets, if you need CNC for the data in the tables — you can write a plugin or snippet, and hang it on the 404 page, which will provide content, if the data is in the table;
    • the
  • do not abuse, for example, calls Ditto, in practice, met with a dozen uncached Ditto call on one page that is terribly slow website, for specific tasks it is better to write specific snippet does nothing superfluous.


Of course, if the expected number of pages on the site in several thousand — before the developer there are already other problems, and if modx and is used only for static content and as a bridge for data in other tables.

Opinion


As you can see, modx can work with a large number of documents related to evolution can be used to create large portals and online shops, though the approach must be smart. Modx is just a tool, how these tools will benefit the developer, depends on the quality of the created resource.
The article does not mention modx revolution, how best to do snippets (in a separate file, or code to store in DB) and much more about modx it's possible to write very much.

Pleasure to read comments or criticism of the article and the code.
Specifically, for the Habra.
Article based on information from habrahabr.ru

Комментарии

Отправить комментарий

Популярные сообщения из этого блога

Address FIAS in the PostgreSQL environment. Part 4. EPILOGUE

Изображение
This is the fourth and last part of the article, which contains examples of creating tables fias_AddressObjects in the database under PostgreSQL, and load data into it about adrenokrome elements of FIAS. After these steps, you can independently test the functions described in first , second , the third parts, copying and running scripts on their creation. The full text of the article is composed of 4 parts. In the first half of this part of the article provides comments to the implementation of the scripts to create tables adresovana elements and populate it with data. The second source scripts. Those readers who are only interested in the source code offered at once to go to Application . the epilogue the where to start You should start with visiting the official website of the Federal Tax Service of the section "Federal information address system" (FIAS) of the page Updates . Upload on your PC latest update or the full database FIAS, if you ...
Далее...

PostgreSQL: Analytics for DBA

Many users of PostgreSQL know that the server at the time of their work collects a variety of statistics, but not everyone knows that it is useful to analyze and how to extract it. This small Toolkit collected some useful queries that give some idea about how to use this "hidden knowledge", which constantly accumulates. These queries can be used to monitor the status of PostgreSQL (manually or with plugins for monitoring systems like Nagios, Cacti or Zabbix), to search for bottlenecks in the server and many other similar tasks. Remember that this is only the tip of the iceberg; in the documentation you can find descriptions of several dozen system ideas that can also be useful for the PostgreSQL administrator. For correct work need to enable Toolkit options stats_block_level and stats_row_level in postgresql.conf and set the parameter of stats_reset_on_server_start in its sole discretion. If each time you restart the PostgreSQL server you change some important configu...
Далее...

Audit Active Directory tools with Powershell releases. Part 1

Изображение
Will begin a series of publications on monitoring of Active Directory. In these articles I will give you the most basic problems and ways of their solution. Based on these data, the functionality can easily be expanded as requirements that You need. Given that now for all OS starting with Windows Server 2003 R2 and Windows XP SP3 Powershell available. I think that this article is useful because it does not require from the administrator of the introduction of any additional costs, i.e. monitoring regular means. So let's begin. Active Directory Monitoring All blog community dedicated to IT you can find many articles on monitoring AD, but... more than 90% of them devoted to the use of third-party apps, most of them facing a certain amount of money, which not every company is willing to give, even if not great. Probably the record for the number of articles is a product from the company NetWrix Corporation. Here and there IT professionals to paint the wonderful p...
Далее...