{"id":7838,"date":"2021-09-06T19:22:30","date_gmt":"2021-09-06T19:22:30","guid":{"rendered":"https:\/\/www.use-snip.com\/kb\/?post_type=ht_kb&p=7838"},"modified":"2021-10-04T21:15:37","modified_gmt":"2021-10-04T21:15:37","slug":"snip-and-host-machine-warnings","status":"publish","type":"ht_kb","link":"https:\/\/www.use-snip.com\/kb\/knowledge-base\/snip-and-host-machine-warnings\/","title":{"rendered":"SNIP and Host Machine Warnings"},"content":{"rendered":"

This article covers the SNIP<\/strong> <\/span>and Host machine<\/em> sections of the automatic error reporting in greater detail. Instructions regarding how to set the trigger thresholds for the needs for your Caster are provided below.\u00a0\u00a0 The SNIP<\/strong><\/span> NTRIP Caster<\/a> provides a range of automatic error reporting mechanisms to alert both the Caster Owner\/Operator and the NTRIP Clients<\/a> when various detrimental operating conditions occur.\u00a0 A higher level overview of these functional sections is provided in this article<\/a>.<\/p>\n

Overview<\/h2>\n

The logic involved in this section deals primarily with SNIP<\/strong><\/span> itself and the machine on which it is hosted, rather than the Base Stations and the User Accounts which connect to it for corrections.\u00a0 The raw data which is used in trigger thresholds is gathered as part of the routine system logs and can be plotted over various time intervals ranging from hours to weeks. [See this article<\/a> for further details on this topic.]\u00a0 Many of the items which are tracked here evolve over periods of many hours or days due to other resource demands on the host machine.<\/p>\n

This section of the dialog consists of seven related logic areas highlighted below.<\/p>\n

\"\"<\/a><\/p>\n

Each functional area can be enabled \/ disabled<\/em> by checking \/ un<\/strong>checking<\/em> the associated check box.\u00a0 Each SetUp<\/strong> button brings up a control dialog where the behavior of that function can be set.\u00a0 For this group of error functions the\u00a0 setup process controls value limits which must be exceeded before a warning eMail is sent out. The Run button causes the logic of the associated function to be run and a summary report to be created. The Run Now<\/strong> button (at the bottom of the dialog) can be used to process all the active functions and send a report.<\/p>\n

Each functional area is discussed in turn.
\n<\/a><\/p>\n

Low Free Memory<\/h3>\n

This test checks the current available free memory on the host machine and reports if it is below the threshold set.\u00a0 This reflects the memory used by all<\/strong> processes and applications on the host machine, not just SNIP<\/strong><\/span>.\u00a0 When the host machine is running low on memory, performance suffers and can cause SNIP<\/strong><\/span> to crash if it is unable to allocate memory it needs. Unlike many desktop applications, SNIP<\/strong><\/span> is constantly servicing data between users and bases.\u00a0 In an NTRIP Caster there is no concept of swapping out large sections of the program during idle times as might be found (for example) in a word processor.\u00a0\u00a0 Generally SNIP<\/strong><\/span> memory use is constant and proportional to the number of Bases connected (various logs grow over the first week of operation). But SNIP<\/strong><\/span> will (temporarily) allocate more memory whenever the user requests large reports, or maps, or loads prior weeks logs, and when other periodic processes must run as background tasks.<\/p>\n

\"\"<\/a><\/p>\n

See also<\/strong><\/em>: High SNIP Memory Use (below) which measures the memory being used by SNIP<\/strong><\/span> itself.
\n<\/a><\/p>\n

Low Free Disk Space, for Data<\/h3>\n

This test checks the current available free disk space on the host machine, using the location where SNIP<\/strong><\/span> \/data<\/strong> is being stored.\u00a0\u00a0\u00a0 The \/data folder location is used to store larger files (typically raw observational data from Base Stations can then be processed into RINEX for further use).\u00a0 You can control where the \/data folder is located with the Log Settings dialog, reached with the menu item Misc<\/strong> \u21d2 Data File Settings…<\/strong> \u00a0 It is common to use a secondary drive to store Base Station data for ease in backing up such files.\u00a0 When the host machine is running low on disk space, GNSS data measurement files may be lost and this can cause SNIP<\/strong><\/span> to crash in extreme cases. Under such conditions, SNIP will continue to service the NTRIP Clients but logging (which also requires disk space) is curtailed.<\/p>\n

\"\"<\/a><\/p>\n

A note for common AWS VM installations<\/strong><\/span>:\u00a0 In the most common AWS deployments (various EC2 medium instances) you will have only one hard drive present with no other external storage.\u00a0 A 30G sized drive is common.\u00a0 In such a deployment the two tests “Low Free Disk Space, for Data<\/strong>”\u00a0 and “Low Free Disk Space, for SNIP \/bin files<\/strong>” are measuring the same drive.
\n<\/a><\/p>\n

Low Free Disk Space, for SNIP \/bin files<\/h3>\n

This test checks the current available free disk space on the host machine, using the location where the SNIP<\/strong><\/span> \/bin<\/strong> folder is located.\u00a0\u00a0 The \/bin folder is where the actual SNIP<\/strong><\/span> application and associated files reside.\u00a0 It is also used to store many parameters and settings used in SNIP<\/strong><\/span> operations (this article on creating a Backup<\/a> provides further details of the various files involved).\u00a0 When the host machine is runs out of disk space in the \/bin folder it will likely cause SNIP<\/strong><\/span> to crash.<\/p>\n

\"\"<\/a><\/p>\n

A note for common AWS VM installations<\/strong>:<\/span>\u00a0 In the most common AWS deployments (various EC2 medium instances) you will have only one hard drive present with no other external storage.\u00a0 A 30G sized drive is common.\u00a0 In such a deployment the two tests “Low Free Disk Space, for Data<\/strong>”\u00a0 and “Low Free Disk Space, for SNIP \/bin files<\/strong>” are measuring the same drive.<\/p>\n

 <\/p>\n

High File Counts<\/h3>\n

This test checks the number of SNIP<\/strong><\/span> related data files which are present on the host machine, in the location (both folder and subfolders) where SNIP<\/strong><\/span> \/data<\/strong> is being stored.\u00a0\u00a0 Over time the number of files can become quite large and this in turn slows down the file system and may (especially for virtual machine deployments) cause storage issues. If your host machine has limited disk space, setting this to a low value can be helpful to remind you to periodically clear off older files.<\/p>\n

\"\"
\n<\/a><\/p>\n

High SNIP Memory Use<\/h3>\n

This test checks the current memory consumed by the SNIP<\/strong><\/span> application on the host machine and reports if it is above the detection threshold set.\u00a0\u00a0 Generally SNIP<\/strong><\/span> memory use is constant and proportional to the number of Bases connected (various logs grow over the first week of operation). Memory use will be observed to grow most rapidly during the first 24 hours after a restart as various buffers reach their allocated sizes.\u00a0\u00a0 But SNIP<\/strong><\/span> will (temporarily) allocate more memory whenever the user requests large reports or maps and when other periodic processes must run as background tasks. [Aside<\/strong><\/span><\/em>: You can also use the Process command<\/a> to check on current memory usage.]<\/p>\n

\"\"<\/a>
\n<\/a><\/p>\n

High SNIP CPU Use<\/h3>\n

This test checks the current CPU processing load used by the SNIP<\/strong><\/span> application on the host machine and reports if it is above the detection threshold set.\u00a0 Except for periodic report generation (which occurs most often at intervals of 4,12, and 24 hours), the CPU use on given SNIP<\/strong><\/span> deployment is fairly constant.\u00a0 This test is primary used is to detect and report when some additional new demand (often from excessive number of bad user connections) is occurring to the SNIP<\/strong><\/span> Caster which requires operator assessment.<\/p>\n

\"\"<\/a>
\n<\/a><\/p>\n

High Host Machine CPU Use<\/h3>\n

This test checks the current CPU processing load used by the ALL <\/strong>applications on the host machine and reports if it is above the detection threshold set. Its primary use is to detect and report when another application on the host machine may be consuming resources that SNIP<\/strong><\/span> needs for efficient Caster operations.<\/p>\n

\"\"<\/a><\/p>\n

A typical Example of use<\/h2>\n

Consider a SNIP<\/strong><\/span> deployment which is not on a dedicated machine. This machine is at times used for other purposes.\u00a0 When this occurs, the other running applications consume process resources.\u00a0 When suitable thresholds have been set, the SNIP<\/strong><\/span> operator can be informed whenever conditions detrimental to the Caster exist.<\/p>\n

Consider the chart below (click to enlarge) showing the average CPU use on the host machine.\u00a0 SNIP<\/strong><\/span> itself is consuming a trivial ~0.4% of the machine, while the total of all processes (including SNIP<\/strong><\/span>) is ~4.3%.\u00a0 This is typical of a machine with only a few active Base Stations and users.\u00a0 At around the time 8:33 the CPU usage increased dramatically.\u00a0 At one point it reached over 80% utilization.\u00a0 [For the purpose of this example, some video clips were loaded and displayed on this machine.]\u00a0\u00a0\u00a0 This continues until about 8:44 when the CPU load is reduced to around 16% where it remained for some time.\u00a0\u00a0 [The video clips were no longer being played but were still consuming memory.]<\/p>\n

This condition would be detected and reported by setting the High SNIP CPU Use<\/strong> or by the High Host Machine CPU Use<\/strong> thresholds. [One could set either of both, and each at their own trigger threshold.]<\/p>\n

\"\"<\/a><\/p>\n

At first glance the above condition may seem transitory. After all, in about ten minutes the CPU use dropped and a considerable spare margin remained for use by SNIP<\/strong><\/span> and other applications.\u00a0 But this is considering only the CPU load which was involved.\u00a0 Often the memory utilization gives a better insight.\u00a0 [And this is why the System Logs dialog<\/a> supports displaying a wide variety of parameters including virtual memory footprints.]<\/p>\n

Below we show the same event, but plotted for the memory footprint used (and displayed over a longer time scale).\u00a0 After the initial moment of increased CPU use we see overall memory on the host machine rise from an original values of ~47% to well over 85% of all memory now being used.\u00a0\u00a0 At this point there is little free memory left for SNIP<\/strong><\/span> to run large report requests or other periodic tasks.\u00a0\u00a0 Note that this condition persists AFTER the time when CPU use is reduced (at ~8:44) to about 9:05 when the application in question was terminated.<\/p>\n

\"\"<\/a><\/p>\n

This condition would be detected and reported by setting the\u00a0Low Free Memory<\/strong>\u00a0 threshold.<\/p>\n

When one or more conditions have been detected, the eMail report issued lists them along with both the current value and the threshold value.\u00a0\u00a0 The key fragment of this report is a table which is shown below. This table is for another machine where SNIP<\/strong><\/span> itself has grown the a large size (a AWS VM deployment which has many Stations).<\/p>\n\n\n\n\n\n\n\n\n\n\n
\u00a0Test For<\/th>\n\u00a0Result<\/th>\n<\/tr>\n
Host Machine Low on Memory<\/b><\/span><\/td>\n71% In use. (85%), Within normal<\/b><\/span> bounds<\/td>\n<\/tr>\n
Data Store Disk Space<\/b><\/span><\/td>\n5,083 MB remains. (1,500 MB), Within normal<\/b><\/span> bounds<\/td>\n<\/tr>\n
Application Disk Space<\/b><\/span><\/td>\n5,083 MB remains. (1,500 MB), Within normal<\/b><\/span> bounds<\/td>\n<\/tr>\n
Data Disk Space File Count<\/b><\/span><\/td>\n157 files. (200), Within normal<\/b><\/span> bounds<\/td>\n<\/tr>\n
SNIP Application, Private Memory use<\/b><\/span><\/td>\n770 MB. Warning!<\/b><\/span> Large Memory Use. (750 MB)<\/td>\n<\/tr>\n
SNIP Application, CPU use<\/b><\/span><\/td>\n20.7% Used. (40.0%), Within normal<\/b><\/span> bounds<\/td>\n<\/tr>\n
Host Machine, CPU use<\/b><\/span><\/td>\n25.4% Used. (40.0%), Within normal<\/b><\/span> bounds<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

The above shows (among other things) that the subject machine is using 71% of its memory, which is still below the trigger point of 85% but that the memory size of SNIP<\/strong><\/span> itself has grown to be 770MB, which is 20MB beyond its allowed size.<\/p>\n

Hint:<\/strong> <\/span>Recall that the checkbox No send if all are normal <\/strong>can be used to disable sending status reports when every enabled category is within its normal bounds.\u00a0 This is useful to reduce unwanted eMail traffic.<\/p>\n","protected":false},"excerpt":{"rendered":"

This article covers the SNIP and Host machine sections of the automatic error reporting in greater detail. Instructions regarding how to set the trigger thresholds for the needs for your Caster are provided below.\u00a0\u00a0 The SNIP NTRIP Caster provides a range of automatic error reporting mechanisms to alert both the […]<\/p>\n","protected":false},"author":13,"comment_status":"open","ping_status":"closed","template":"","format":"standard","meta":{"_exactmetrics_skip_tracking":false,"_exactmetrics_sitenote_active":false,"_exactmetrics_sitenote_note":"","_exactmetrics_sitenote_category":0,"footnotes":""},"ht-kb-category":[418],"ht-kb-tag":[510,487,509,201],"class_list":["post-7838","ht_kb","type-ht_kb","status-publish","format-standard","hentry","ht_kb_category-reports","ht_kb_tag-automatic-error-reporting","ht_kb_tag-email","ht_kb_tag-monitoring","ht_kb_tag-reports"],"_links":{"self":[{"href":"https:\/\/www.use-snip.com\/kb\/wp-json\/wp\/v2\/ht-kb\/7838","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.use-snip.com\/kb\/wp-json\/wp\/v2\/ht-kb"}],"about":[{"href":"https:\/\/www.use-snip.com\/kb\/wp-json\/wp\/v2\/types\/ht_kb"}],"author":[{"embeddable":true,"href":"https:\/\/www.use-snip.com\/kb\/wp-json\/wp\/v2\/users\/13"}],"replies":[{"embeddable":true,"href":"https:\/\/www.use-snip.com\/kb\/wp-json\/wp\/v2\/comments?post=7838"}],"version-history":[{"count":24,"href":"https:\/\/www.use-snip.com\/kb\/wp-json\/wp\/v2\/ht-kb\/7838\/revisions"}],"predecessor-version":[{"id":7969,"href":"https:\/\/www.use-snip.com\/kb\/wp-json\/wp\/v2\/ht-kb\/7838\/revisions\/7969"}],"wp:attachment":[{"href":"https:\/\/www.use-snip.com\/kb\/wp-json\/wp\/v2\/media?parent=7838"}],"wp:term":[{"taxonomy":"ht_kb_category","embeddable":true,"href":"https:\/\/www.use-snip.com\/kb\/wp-json\/wp\/v2\/ht-kb-category?post=7838"},{"taxonomy":"ht_kb_tag","embeddable":true,"href":"https:\/\/www.use-snip.com\/kb\/wp-json\/wp\/v2\/ht-kb-tag?post=7838"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}