السبت، 31 يناير 2009

Open redirect URLs: Is your site being abused?

No one wants malware or spammy URLs inserted onto their domain, which is why we all try to follow good security practices. But what if there were a way for spammers to take advantage of your site, without ever setting a virtual foot in your server?

There is, by abusing open redirect URLs.

Webmasters face a number of situations where it's helpful to redirect users to another page. Unfortunately, redirects left open to any arbitrary destination can be abused. This is a particularly onerous form of abuse because it takes advantage of your site's functionality rather than exploiting a simple bug or security flaw. Spammers hope to use your domain as a temporary "landing page" to trick email users, searchers and search engines into following links which appear to be pointing to your site, but actually redirect to their spammy site.

We at Google are working hard to keep the abused URLs out of our index, but it's important for you to make sure your site is not being used in this way. Chances are you don't want users finding URLs on your domain that push them to a screen full of unwanted porn, nasty viruses and malware, or phishing attempts. Spammers will generate links to make the redirects appear in search results, and these links tend to come from bad neighborhoods you don't want to be associated with.

This sort of abuse has become relatively common lately so we wanted to get the word out to you and your fellow webmasters. First we'll give some examples of redirects that are actively being abused, then we'll talk about how to find out if your site is being abused and what to do about it.

Redirects being abused by spammers

We have noticed spammers going after a wide range of websites, from large well-known companies to small local government agencies. The list below is a sample of the kinds of redirect we have seen used. These are all perfectly legitimate techniques, but if they're used on your site you should watch out for abuse.

  • Scripts that redirect users to a file on the server—such as a PDF document—can sometimes be vulnerable. If you use a content management system (CMS) that allows you to upload files, you might want to make sure the links go straight to the file, rather than going through a redirect. This includes any redirects you might have in the downloads section of your site. Watch out for links like this:
example.com/go.php?url=
example.com/ie/ie40/download/?

  • Internal site search result pages sometimes have automatic redirect options that could be vulnerable. Look for patterns like this, where users are automatically sent to any page after the "url=" parameter:
example.com/search?q=user+search+keywords&url=

  • Systems to track clicks for affiliate programs, ad programs, or site statistics might be open as well. Some example URLs include:
example.com/coupon.jsp?code=ABCDEF&url=
example.com/cs.html?url=

  • Proxy sites, though not always technically redirects, are designed to send users through to other sites and therefore can be vulnerable to this abuse. This includes those used by schools and libraries. For example:
proxy.example.com/?url=

  • In some cases, login pages will redirect users back to the page they were trying to access. Look out for URL parameters like this:
example.com/login?url=

  • Scripts that put up an interstitial page when users leave a site can be abused. Lots of educational, government, and large corporate web sites do this to let users know that information found on outgoing links isn't under their control. Look for URLs following patterns like this:
example.com/redirect/
example.com/out?
example.com/cgi-bin/redirect.cgi?

Is my site being abused?

Even if none of the patterns above look familiar, your site may have open redirects to keep an eye on. There are a number of ways to see if you are vulnerable, even if you are not a developer yourself.

  • Check if abused URLs are showing up in Google. Try a site: search on your site to see if anything unfamiliar shows up in Google's results for your site. You can add words to the query that are unlikely to appear in your content, such as commercial terms or adult language. If the query [site:example.com viagra] isn't supposed to return any pages on your site and it does, that could be a problem. You can even automate these searches with Google Alerts.

  • You can also watch out for strange queries showing up in the Top search queries section of Webmaster Tools. If you have a site dedicated to the genealogy of the landed gentry, a large number of queries for porn, pills, or casinos might be a red flag. On the other hand, if you have a drug info site, you might not expect to see celebrities in your top queries. Keep an eye on the Message Center in Webmaster Tools for any messages from Google.

  • Check your server logs or web analytics package for unfamiliar URL parameters (like "=http:" or "=//") or spikes in traffic to redirect URLs on your site. You can also check the pages with external links in Webmaster Tools.

  • Watch out for user complaints about content or malware that you know for sure can not be found on your site. Your users may have seen your domain in the URL before being redirected and assumed they were still on your site.


What you can do

Unfortunately there is no one easy way to make sure that your redirects aren't exploited. An open redirect isn't a bug or a security flaw in and of itself—for some uses they have to be left fairly open. But there are a few things you can do to prevent your redirects from being abused or at least to make them less attractive targets. Some of these aren't trivial; you may need to write some custom code or talk to your vendor about releasing a patch.

  • Change the redirect code to check the referer, since in most cases everyone coming to your redirect script legitimately should come from your site, not a search engine or elsewhere. You may need to be permissive, since some users' browsers may not report a referer, but if you know a user is coming from an external site you can stop or warn them.

  • If your script should only ever send users to an internal page or file (for example, on a page with file downloads), you should specifically disallow off-site redirects.

  • Consider using a whitelist of safe destinations. In this case your code would keep a record of all outgoing links, and then check to make sure the redirect is a legitimate destination before forwarding the user on.

  • Consider signing your redirects. If your website does have a genuine need to provide URL redirects, you can properly hash the destination URL and then include that cryptographic signature as another parameter when doing the redirect. That allows your own site to do URL redirection without opening your URL redirector to the general public.

  • If your site is really not using it, just disable or remove the redirect. We have noticed a large number of sites where the only use of the redirect is by spammers—it's probably just a feature left turned on by default.

  • Use robots.txt to exclude search engines from the redirect scripts on your site. This won't solve the problem completely, as attackers could still use your domain in email spam. Your site will be less attractive to attackers, though, and users won't get tricked via web search results. If your redirect scripts reside in a subfolder with other scripts that don't need to appear in search results, excluding the entire subfolder may even make it harder for spammers to find redirect scripts in the first place.



Open redirect abuse is a big issue right now but we think that the more webmasters know about it, the harder it will be for the bad guys to take advantage of unwary sites. Please feel free to leave any helpful tips in the comments below or discuss in our Webmaster Help Forum.

Written by Jason Morrison, Search Quality Team


الأربعاء، 28 يناير 2009

Add on to AdSense

Do you regularly work with your webpages and AdSense implementation, tinkering with HTML or PHP and creating images and code on the fly? If you do, Firefox add-ons can help streamline the process of creating webpages. Here are some in particular that you may find useful:

ColorZilla
This extension tells you which RGB or hex color you're looking at, to help you make sure you created that logo for your business with just the right shade of blue, for instance. The tool also creates custom color palettes while you're browsing, so you can use them in your designs.

MeasureIt
Like the name says, use this add-on to measure the width and height in pixels of any element you see on a webpage. It's very simple to use, and you can define how much space you have left for that AdSense ad unit on the right-side. :)

IE View
Do you frequently use Internet Explorer to check how your website renders on that browser? This add-on allows you to view the way any page would look if it were opened in IE, without the hassle of opening another browser. You can also see pages that aren't Firefox-friendly much more easily.

WebDeveloper toolbar
This all-in-one toolbar gives you quick control over things like JavaScript display, form and CSS elements, screen resizing (so you know what your website looks like in smaller resolutions), HTML validation, and much more.

Hopefully, we'll soon have a similar set of add-ons for Chrome, and we'll be sure to share them with our readers.

What are your favorite add-ons for web developing? Leave us a comment below.



الجمعة، 23 يناير 2009

Ads back up on Blogger

If you have a Blogger site, some of you may have noticed that your ad units began showing public service announcements (PSAs) in the last nine hours. This was due to a technical issue that we've now identified and resolved. As a result, you should now begin seeing paid ads on your pages again.

If you're still displaying PSAs after 24 hours, we recommend reviewing our Help Center and using our PSA troubleshooter.

Thanks to everyone who helped report this issue, and we appreciate your patience.



الخميس، 22 يناير 2009

Year in Review

2008 was another great year for the Webmaster Central team. We experienced tremendous user growth with our blogs (97% increase in monthly pageviews), Help Center (25%), Help Forums (225%), and Webmaster Tools (35%). We would like to welcome our new users that joined us in '08, and thank our loyal and passionate user base that have been with us for the last couple of years. We focused on two basic goals for 2008, and here's how we think we did:

Goal #1: Educate and grow our webmaster community
  • We had our first ever online webmaster chat in February '08 to answer your top questions, and followed it up with three more. They have been incredibly successful, and we're planning for more this year.
  • We'd like to send a special thank you to our Bionic Posters, who have played a huge part in supporting our growing community.
  • Localization has been a big focus for us, so we launched our blog and Help Center in additional languages, and made Webmaster Tools available in 40 languages. We hope this makes it easier for people in other parts of the world to adopt our tools and gain a better understanding of how search works.
  • We launched a new Help Forum in English and Polish, with a broader rollout planned in other languages this year.
  • Our SEO starter guide was released and it has been one of our most successful articles to date.
  • We placed an emphasis on sharing material via YouTube and created seven video series totaling two hours of content. We kicked off '09 with a bang on the video front with Matt's "Virtual Blight" presentation.
Goal #2: Iterate early and often on Webmaster Tools
Thank you once again and we hope for another exciting and eventful year!



الثلاثاء، 20 يناير 2009

Update on US tax forms for 2008

This is a friendly reminder that we're currently preparing and mailing tax forms to eligible U.S. publishers and will be sending them out by the end of January. Please keep in mind that not all U.S. AdSense publishers will receive a tax form from Google.

How do you know whether to expect a tax form? We'll send you one if:
  • You submitted a Form W-9, are not a corporation, and were paid at least US $600 in 2008, OR
  • You indicated that you are subject to backup withholding and had taxes withheld
If you qualify for a tax form, you can expect it to arrive at the address listed in your account by early February. We recommend checking your account to ensure that your mailing address is up-to-date; you can make any necessary updates by following the instructions in our Help Center. Please note that payments dated in 2008 will be reported in 2008. This means that unpaid earnings from 2008 that are rolled over to 2009 (for example, December 2008 earnings paid in January 2009) will not be included.

If you won't be receiving a tax form, but you still have questions on how to report the payments you received from AdSense, please consult your local tax advisor.



الخميس، 15 يناير 2009

Adding a social playlist to your site

As you're building your site, you may be looking for a simple way to provide fresh content that captures the attention of first time visitors and loyal users alike. They say that music brings people together, so what better way to engage your visitors than by inviting them to help build a unique, collaborative soundtrack for your website? Now, social application creator iLike has built a special version of their social playlist gadget for sites using Google Friend Connect.

Visitors can add their favorite songs
iLike's playlist gadget lets you and your visitors shape the site's "musical footprint" as a group. With this application, anyone visiting your website can listen to songs on the playlist, and if they sign in using Friend Connect, they can add their own favorites to the list. Of course, you can also add songs to the playlist, and as the site administrator, you have the ability to remove songs or change the order.

If you already have Friend Connect running on your website, you can add some musical flair in a matter of minutes with just a few clicks. Sign in at www.google.com/friendconnect, click "Social Gadgets," and you'll find the iLike "Playlist gadget" in the gallery.


Select the "Playlist gadget," and Friend Connect will automatically generate a snippet of code for you to copy-and-paste into your website's HTML. While you're there, you may also consider adding the "Wall gadget"—music can be a great conversation starter!

This iLike gadget is fully integrated with your existing Friend Connect account, so you can edit your website's playlist, moderate wall posts, and manage membership all from a single interface.

Like all of the social applications that work with Friend Connect, iLike's application is built using OpenSocial, and it's a great example of how a social application can foster a sense of community around a website. Any site using Friend Connect can host gadgets created by the OpenSocial developer community.

If you're a site owner who wants to begin adding social features to your website, visit Google Friend Connect. No programming is required!

If you're a developer interested in building a social application to run on the tens thousands of websites that are now using Google Friend Connect, learn more at www.opensocial.org.



Talking AdSense optimization in Google Ad Manager

It can be tough to sell advertising in today's economy. It can be even tougher to figure out how to maximize revenue for each ad impression. Google Ad Manager, our hosted ad serving and management solution for publishers with small direct sales teams, was built to address these issues. Ad Manager helps publishers maximize the value of their ad impressions while reducing ad serving costs.

We enlisted the help of Nandu Ramani, Engineering Lead on Ad Manager, to talk about one of Ad Manager's features that helps publishers maximize the value of their ad impressions: AdSense price optimization.

What is the AdSense price optimization feature in Ad Manager?

Many publishers don't sell all of their ad inventory. In these situations, publishers might not serve any ads or might serve less valuable house ads, therefore losing potential earnings. The AdSense price optimization feature in Ad Manager provides an automated solution so publishers will always have an ad to serve in an undersold situation.

We also wanted to make sure that when a publisher runs multiple ad networks they are always showing the most valuable ads. For certain individual impressions, AdSense can provide the highest paying ad. When that's the case, an AdSense ad shows. When that's not the case, an ad from the highest paying alternative network will be shown.

How does the price optimization feature work?

In order for AdSense to compete against other ad networks, a publisher must manually enter a CPM for each configured network. We use the CPM entered to determine in real time, on a per impression basis, whether or not an AdSense ad will pay a publisher more. If the AdSense eCPM is greater than the CPM value entered for competing networks, then an AdSense ad will be shown. Additionally, AdSense will never compete with a publisher's directly-sold inventory. To enable the price optimization feature, all a publisher has to do is check a box when setting up inventory.

As a publisher, how much will I earn using AdSense price optimization?

It's hard to predict; the best way to find out is to opt your ad slots into AdSense price optimization and see how AdSense performs for you. With AdSense price optimization, Google will always serve the highest paying AdSense ad available, and will never lower the price of the winning ad, or reduce your earnings from it.

When should I use the AdSense price optimization feature?

We suggest you opt all of your ad slots into AdSense price optimization. AdSense ads will only appear if they're able to pay you more than the alternatives, so there's no risk of losing revenue.

We also recommend that you opt your premium placements into placement targeting so AdWords advertisers may specifically choose to bid for space on your website.

Sounds good. How do I get started?

If you already have an Ad Manager account, go into the inventory tab in your account. For each ad slot where you want to enable AdSense price optimization, click on the name of the ad slot, check the 'Maximize revenue of unsold and remnant inventory with AdSense' checkbox, and click 'Save.'

If you don't already have an account, get started today at http://www.google.com/admanager. Then, when you're setting up your inventory, make sure to opt all of your ad slots into AdSense price optimization.

For more information about AdSense in Ad Manager, check out the following video.



Seamless verification of Google Sites and Blogger with Webmaster Tools

Verifying that you own a site is the first step towards accessing all of the great features Webmaster Tools has to offer, such as crawl errors and query statistics. The Google Sites and Blogger teams have worked hard to make site verification as simple as possible. In the following videos, I'll walk you through how to verify sites created in Google Sites and Blogger.

Google Sites:


Blogger:


These videos are available in our Help Center if you have additional questions about verifying a Google Site or Blogger blog with Webmaster Tools. And as always, you can find me and many other Googlers and webmasters in our Webmaster Help Forum.



الثلاثاء، 13 يناير 2009

Asking Dave Taylor about AdSense

AskDaveTaylor.com offers tech support Q&A on subjects ranging from mp3 players to Linux to AdSense. We recently chatted with founder Dave Taylor about his site and his AdSense experience.

Inside AdSense: Where did the idea for your 'Ask Dave Taylor' site come from?

Dave Taylor: There's a great backstory, actually. I've written twenty different books on various business and technical topics, including Teach Yourself Unix in 24 Hours and Creating Cool Web Sites. Each time I'd publish, I would be sure to include my email address and other contact information. Problem was, people would send me email with questions. Lots of email with questions.

Over time I found myself answering the same questions again and again and realized that there had to be a better way for readers to search through an archive of already answered questions. I tried an online discussion forum, but it didn't really work very well (though it did give me an excuse to write my own bbs system from scratch, but that's another story!).

Then early in 2003 this "weblog" thing started to gain a bit of traction. When I first saw how it was built upon the concept of an author writing entries and others being able to add their comments, I realized that it could be ideal for my needs.

IA: Why did you join the AdSense program?

DT: As a businessperson, I had always viewed my website as a cost center. I mean, you had to pay for hosting, you had to pay for graphic design, you had to pay for Internet connectivity, etc. That was just my mindset. It was a marketing expense and its purpose was lead generation for my consulting and book sales.

In mid-2003 my friend told me about this "AdSense thing" and said that he'd been experimenting with it and making some money. So I finally decided that I'd try putting some adverts on my site (I'd been on the Web since 1996 but never had any adverts on my sites until that point). That first month I made more than I expected by simply adding the AdSense adverts to my pages and was surprised as heck. Then it started to grow...

That's when it hit me, that my website was becoming a profit center for my business, not a cost center. I began to pay more attention to the site and published new content on a more regular basis. Within a few months I was earning enough to pay my mortgage, and today my website, and specifically Google AdSense, is a primary revenue stream for my entire company.

IA: Can you talk a little about your experience with optimizing your ads?

Once I began working with AdSense in earnest, I began to wonder how ad placement, size, color, and design would affect earnings, and how to balance my desire to offer a splendid user experience with the need to simultaneously maximize revenue.

Enter A/B testing. I read and talked with many AdSense publishers, tried what they suggested and what had worked for them, fiddled with my own ideas, and generally tried every variation I could imagine to see if I could improve the click-through-rate of my ad blocks. The greatest boosts I saw in clickthrough rate were when I moved the advert into the middle of my articles, when I made sure it had the same color background as the material around it, and when there wasn't a solid border or other visual element to make the ad stand out from the surrounding content.


Truth be told, I've also paid close attention to the sites profiled on the AdSense blog, looking at how they integrated ads into their own design and trying to emulate their successful techniques on my own site.

IA: Glad to hear you used the blog! Any other optimization tips for our readers?
  1. Focus on generating really good content that meets real user needs.
  2. Design your blog so that there are minimal distractions for the user.
  3. Wrap your blog entry around the Google ad unit and put the ads where users will see them, though make sure you have them visually distinct from your content: trying to trick readers into clicking on ads is a definite no-no and anti-reader too.
IA: Thanks for the interview, Dave, and good luck with your site!

Do you also have an AdSense success story to share? Let us know.



A new Google Sitemap Generator for your website

It's been well over three years since we initially announced the Python Sitemap generator in June 2005. In this time, we've seen lots of people create great third-party Sitemap generators to help webmasters create better Sitemap files. While most Sitemap generators either crawl websites or list the files on a server, we have created a different kind of Sitemap generator that uses several ways to find URLs on your website and then allows you to automatically create and maintain different kinds of Sitemap files.

Google Sitemap Generator screenshot of the admin console

About Google Sitemap Generator


Our new open-source Google Sitemap Generator finds new and modified URLs based on your webserver's traffic, its log files, or the files found on the server. By combining these methods, Google Sitemap Generator can be very fast in finding these URLs and calculating relevant metadata, thereby making your Sitemap files as effective as possible. Once Google Sitemap Generator has collected the URLs, it can create the following Sitemap files for you:

In addition, Google Sitemap Generator can send a ping to Google Blog Search for all of your new or modified URLs. You can optionally include the URLs of the Sitemap files in your robots.txt file as well as "ping" the other search engines that support the sitemaps.org standard.

Sending the URLs to the right Sitemap files is simple thanks to the web-based administration console. This console gives you access to various features that make administration a piece of cake while maintaining a high level of security by default.

Getting started


Google Sitemap Generator is a server plug-in that can be installed on both Linux/Apache and Microsoft IIS Windows-based servers. As with other server-side plug-ins, you will need to have administrative access to the server to install it. You can find detailed information for the installation in the Google Sitemap Generator documentation.

We're excited to release Google Sitemap Generator with the source code and hope that this will encourage more web hosters to include this or similar tools in their hosting packages!

Do you have any questions? Feel free to drop by our Help Group for Google Sitemap Generator or ask general Sitemaps question in our Webmaster Help Forum.



Preventing Virtual Blight: my presentation from Web 2.0 Summit

One of the things I'm thinking about in 2009 is how Google can be even more transparent and communicate more. That led me to a personal goal for 2009: if I give a substantial conference presentation (not just a question and answer session), I'd like to digitize the talk so that people who couldn't attend the conference can still watch the presentation.

In that spirit, here's a belated holiday present. In November 2008 I spoke on a panel about "Preventing Virtual Blight" at the Web 2.0 Summit in San Francisco. A few weeks later I ended up recreating the talk at the Googleplex and we recorded the video. In fact, this is a "director's cut" because I could take a little more time for the presentation. Here's the video of the presentation:



And if you'd like to follow along at home, I'll include the actual presentation as well:



You can also access the presentation directly. By the way thanks to Wysz for recording this not just on a shoestring budget but for free. I think we've got another video ready to go pretty soon, too.



الأربعاء، 7 يناير 2009

Although I understand Google's stance on this, I w...

Although I understand Google's stance on this, I would recommend not referring to it as a "service" when Google is not doing anything.

I propose a modification of this concept. Google already tracks clicks on its search results, so why not take this a step further? Each time a user clicks a search result, Google POSTs the user's IP address to the page in question, which would then allow that IP to view full pages for the next X minutes (something short, 5-10). After that point, that IP would be shown the usual registration page for Y amount of time (from 1-24 hours). In this fashion, the user could be verified as having visited from Google results (only allow the POST from Google IPs), and it would provide users with full results for a short time (enough to answer a simple question, since sometimes the first page does not have a suitable answer) but not cause a site's premium content to be freely accessible at all times using simple referrer spoofing, which could easily be automated via a small addon in any of the major browsers.

In addition, this would indeed actually be a Google service, since Google would be providing a form of authentication.