Antivirus Performance report

[wonderwrench]

Any idea when we will see an update?
Thanks
Bill

[David (PassMark)]

We are running some tests at the moment, as the 2011 products appear on the market.

Not sure about date, probably sometime in the next month or so.

[wonderwrench]

I see the August Antivirus report just came out, nice! Question, how can you compare results from previous reports? You are now testing using windows 7 64. Previously the tests were run on XP 32 or Vista 32 though I can't prove it because the link for the previous test does not link to the previous test. It links to the new test. The hardware are you using now is the latest cream of the crop stuff. From memory you were using a laptop of some sort before. IMO the new report should only include products tested during the same period, same OS and hardware. IMO the results will not pull much weight until old results are either gotten rid of or the products are retested.

Other thoughts: Many newer products build a cache or database of files previously scanned. Some only build the database during a manual scan. Some build the database through the resident protection also. Some programs have persistent cache some have transient cache, some have both. Programs that use these methods are going to be harder to test and get accurate results. Manual scans will be much slower when the database is being built than it will be once the data base is built. I would guess most of the inconsistency can be gotten around by running a complete system scan before any performance metrics are measured. I see your manual scans are being run by right click context menu. IMO all products should scan every file in the folder on right click.
If a given product uses caching it is poorly thought out or looking for better benchmark performance. IMHO a full scan from within the programs GUI using default settings would be the only fair way to test scan performance because most users are going to scan using this method. I would think this would cover a scheduled scan also.

I see Avast 5 Internet Security is missing from the test. Avast 5 just missed your last report so I would have thought it would surely make this report. I hope in future reports that all available free versions of Antivirus will be tested because of the shear number of people that use them. Avast and Avira have more users than all other products combined paid or free.

I was waiting for this report because I have an older laptop running XP pro and wanted to know which AV used the least resources. Looks like I need to do some testing myself. BTW I know it is a huge undertaking just testing in windows 7 but XP still has the largest user base. In a perfect world both XP and windows 7 would be tested.

I do thank you for all your hard work. I just wish I could use it as part of the process of choosing a security product.

Thanks
Bill

[David (PassMark)]

how can you compare results from previous reports?

We aren't. The previous report was using Vista. But we aren't using Vista results in the recent report. We also tested on Win7 last year, but we didn't do a separate public report. So the same O/S and hardware was used for all tests.

though I can't prove it because the link for the previous test does not link to the previous test

I'll check this.

Many newer products build a cache or database of files previously scanned

Correct. So the first scan is slow, and subsequent scans are often faster, depending on the product. But we do multiple scans and average the initial result with all subsequent results. So both initial & subsequent performance influence the result. But the real life situation is more complex as it must take into account signature and engine updates. A signature update can invalidate part or all of a cache. And you get signature updates daily with many products.

Avast and Avira have more users than all other products combined paid or free

I doubt this is the case. I know people using the free AVG and free Microsoft security Essentials, but don't know a single person using Avast and Avira. Clearly they are popular, but I don't think they are the market leaders.

BTW I know it is a huge undertaking just testing in windows 7 but XP still has the largest user base

In terms of the machines we see coming in as benchmarks, we reached a cross over point a month or two back. XP is now a minority and dropping fast. Vista is a small minority. Win7 is now the majority. In 12 months XP will be only slightly more popular than Windows NT4

[wonderwrench]

I doubt this is the case. I know people using the free AVG and free Microsoft security Essentials, but don't know a single person using Avast and Avira. Clearly that are popular, but I don't think they are the market leaders.

I should have said the two together have more market share than the better known AV companies do combined. Different parts of the world use different products. People I know in the US either use the free software from their internet provider Norton/McAfee or free software Avast/Avira.

I do not know how accurate it is but have a look at this chart. Avast has about 20% market share. Avira has about 11%. Symantec has about 10%. Microsoft 9%. AVG has under 9%. McAfee has under 8%.

In terms of the machines we see coming in as benchmarks, we reached a cross over point a month or two back. XP is now a minority and dropping fast. Vista is a small minority. Win7 is now the majority. In 12 months XP will be only slightly more popular than Windows NT4

Windows 7 just passed Vista in the last month or so. I think the data being collected by PT is skewed to favor W7. Most of the people that use PT are power users and closer to the cutting edge than the average user. New PC's need a new OS, most likely windows 7 64. > 4 gig memory support and SSD TRIM support etc. etc. OS market share as of July of 2010. I think your statistics are like looking into the future so they are useful but not accurate at this time. I would guess it will take windows 7 two to three years, maybe longer to have a larger user base than XP.

Bill

[wonderwrench]

Correct. So the first scan is slow, and subsequent scans are often faster, depending on the product. But we do multiple scans and average the initial result with all subsequent results. So both initial & subsequent performance influence the result. But the real life situation is more complex as it must take into account signature and engine updates. A signature update can invalidate part or all of a cache. And you get signature updates daily with many products.

I would think if a product shows this behavior it should be mentioned in the results. Better yet would be to do testing with the caching on and off and report both results.

It would be nice to see program uninstall testing. Time and thoroughness of the uninstall. I bet several venders uninstallers will either out and out fail, leave crap all over the place or worse break something.

[David (PassMark)]

Better yet would be to do testing with the caching on and off...

We haven't seen any products that allow it to be turned off. Can't even reboot to clear it a lot of the time.

I bet several venders uninstallers will either out and out fail...

There is a bunch of additional stuff that could be looked at, and there are definitely uninstallers that don't work. Saw one last week that protected parts of the registry from modification, for "security" reasons, but then it's own uninstaller would fail to work because it couldn't remove entries from to the registry. So the product could not be uninstalled.

With all these reports that we do the customer purchasing the report selects the the products they want to compare against and the criteria they want to look at.

[wonderwrench]

We haven't seen any products that allow it to be turned off. Can't even reboot to clear it a lot of the time.

Avast 5 allows quite an elaborate set of options.

Default quick scan


Default full scan


Default file system shield

With all these reports that we do the customer purchasing the report selects the the products they want to compare against and the criteria they want to look at.

Yup Symantec paid for the reports and its too bad they had you skip 30% of the market. In effect Symantec lost the possibility of gaining 30% more market share. You can't sell to owners of products not tested. By not testing all major products the report can't pull the weight Symantec wants it to. I hope they re-think their philosophy. NIS looks to have greatly improved as its far less of a hog than it was not long ago.
Maybe the report is really for internal use only?

The testing you have been reporting has not gone unseen. Power users have been looking for data like this for years. Previously you could get a fair idea how good a given products protection was but had no idea how much of a hog it was without testing it your self (av-text.org/av-comparatives.org etc). Now using your testing we can compare performance while also looking at protection and make a decision as to what to buy/use. Most people have no idea what to use. They either use what their ISP provides or ask someone that they trust as to what they should use.

Bill

[Karen]

Hi Wonderwrench!

I do not know how accurate it is but have a look at this chart. Avast has about 20% market share. Avira has about 11%. Symantec has about 10%. Microsoft 9%. AVG has under 9%. McAfee has under 8%.

I've seen that chart linked from other security forums, actually! Though interesting, I believe that the chart isn't an accurate representation of the market at large as it's only reporting on people who've made use of the OESIS Framework. I'm inclined to believe that on average, people who are using the OESIS SDK would be more technical than 'regular home users' and the results would be somewhat skewed as a result. From my personal (admittedly anecdotal) experience, technical users tend to be more comfortable with using Antivirus only products.

Avast 5 allows quite an elaborate set of options.

Thanks for that. It's applaudable that Avast! provides their users that amount of control over search behavior. Though, Avast! is but one product amidst several dozen who don't yet have this level of configuration; and taking the initial scan time several times would require a clean installation for most products.

Yup Symantec paid for the reports and its too bad they had you skip 30% of the market. In effect Symantec lost the possibility of gaining 30% more market share. You can't sell to owners of products not tested. By not testing all major products the report can't pull the weight Symantec wants it to. I hope they re-think their philosophy. NIS looks to have greatly improved as its far less of a hog than it was not long ago.
Maybe the report is really for internal use only?

The current release of the report (Report 1) is very limited in terms of products. Generally speaking, the first report compares new major versions of Internet Security products which are available at the time of testing. At that stage (June), very few vendors had released their Internet Security products for 2011 and thus it wasn't possible for us to have collected results for these as-yet unreleased products.

We intend to have results for the popular, free Antivirus only products (such as avast! Free AV, Avira and Microsoft Security Essentials) in Report 2, as well as additional results for Internet Security products that were released in the interim.

By the final version of the report, I believe we should have results for most of the global consumer Antivirus market.

What were the products that captured the 30% market share that you believe Symantec are leaving out?

- Karen

[David (PassMark)]

There is another interesting report from May 2010 here,
http://www.pandainsight.com/es/wp-co...-17-May-10.pdf
On page 7 in the pie chart, it puts Avast at 5% market share. Avira doesn't rate at all being less than 2%.

But the table under that (showing numbers from claims made by the companies themselves) paint a different picture. Kaspersky seem to have claimed around 25% of the market. Which I am sure is not true in real life.

[wonderwrench]

What were the products that captured the 30% market share that you believe Symantec are leaving out?

If you believe the chart I linked to Avast and Avira has 30% between the two.

[wonderwrench]

There is another interesting report from May 2010 here,
http://www.pandainsight.com/es/wp-co...-17-May-10.pdf
On page 7 in the pie chart, it puts Avast at 5% market share. Avira doesn't rate at all being less than 2%.

But the table under that (showing numbers from claims made by the companies themselves) paint a different picture. Kaspersky seem to have claimed around 25% of the market. Which I am sure is not true in real life.

The chart many be accurate as its only the US market. Norton and McAfee are mostly North America. Avast, Avira and Kaspersky have huge market share in Europe. Avast free and Avira free are huge in 3rd world countries.
Some where I read Avast figures they are protecting over 110 million PC's.
Figures are claimed to be obtained from their update servers. Each PC can be individually ID'd so the servers keep track of the totals for each day, week, month etc.

Bill