According to Kottke, up until yesterday, the robots.txt file (the file on a web site which tells the search engines which pages to disallow) contained approx. 2400 exclusions including the following:

User-agent: *
Disallow: /cgi-bin
Disallow: /search
Disallow: /query.html
Disallow: /omb/search
Disallow: /omb/query.html
Disallow: /expectmore/search
Disallow: /expectmore/query.html
Disallow: /results/search
Disallow: /results/query.html
Disallow: /earmarks/search
Disallow: /earmarks/query.html
Disallow: /help
Disallow: /360pics/text
Disallow: /911/911day/text
Disallow: /911/heroes/text
Disallow: /911/messages/text
Disallow: /911/patriotism/text
Disallow: /911/patriotism2/text
Disallow: /911/progress/text
Disallow: /911/remembrance/text
Disallow: /911/response/text
Disallow: /911/sept112002/text
Disallow: /911/text
Disallow: /ConferenceAmericas/text
Disallow: /GOVERNMENT/text
Disallow: /africanamerican/text
Disallow: /africanamericanhistory/text
Disallow: /agencycontact/text
Disallow: /appointments/text
Disallow: /avian/text
Disallow: /avianflu/text
Disallow: /bioshield/text
Disallow: /birdflu/text
Disallow: /blackhistory/text
Disallow: /budget/text

So, Google, with the Bush administration, do not ‘expect more’… Being from across the pond, it is not my place to comment as to whether or not one was able to expect more from Bush and what was actually delivered. However, I think the world is excited by the prospect and the potential promise of a truly multicultural president at the highest auspices of the government. I digress.

Bush’s robot.txt says disallow information on 911 heroes, messages, patriotism - which to me seems sad. Ok, maybe they dont want to rank on the search engines for words within the messages left but isn’t this an important legacy? 911 is about many things, which is fiercely debated thoughout the web, but one of the biggest take-aways is the demonstration of the spirit of the American people which words struggle to sum up: courage, tenacity, kindness, selflessness, support.

Google, do not include information on our ‘bioshield’ - intriguing- ‘avianflu’ or our ‘budget’. Need we say more? But with the new dawn in US politics comes the launch of a new robots.txt and new ‘accessibility’ for the search engines.

Apparently Barrack Obama’s team, perhaps in an effort towards transparency(?), has removed all of these exclusions and has a robots.txt file which allows Googlebot to visit all of the files on its site, apart from the /include/ file. Interesting.

Looking at other people covering this on the web, from Jason’s post, I smiled at part of Ron’s comment on shimonsandler.com: “…would I need to disallow robots.txt file in my robots.txt file so the robots.txt file isn’t indexed?” which was possibly a serious comment but amusing nonetheless.

Thank you so much to Web Accessibility Guru , Ian, who forwarded this on to me this morning and it tickled me so much I had to commit ‘fingers to keys’ and share this one straight away.

Looking forward to the new edition of Build Your Own Web Site, Ian.

With an excellent level of support from Microsoft Paid and Editorial departments - including Nathan Buggia and Tor Crockatt, plus European and U.S. Search Professionals, the conference was highly anticipated by all. With three tracks running simultaneously this year for all slots, it indicated just how much demand there is by information-hungry companies and search professionals. Chris Sherman, beautifully assisted by the lovely Vanessa Fox, guided the event smoothly from the kick-off session with Microsoft discussing its new offerings through to Web Site panel conclusion.

Much of SMX London 2008 is covered on numerous blogs, including the ProperoOnline Marketing Blog. For me, I particularly enjoyed the link building sessions with the eloquent Brian Turner, the dynamic Dixon Jones and the very smart & amusing Matt Paines and the various International / European SEO sessions with Andy Atkins-Kruger, the lovely Lisa D, Duncan Morris et al.

With many opportunities to meet people and network, great organisation, truly superb sessions and a sound central London location, the SMX really hit it on the nail with this one. Looking forward to the next SMX in Europe, in Munich, April 2009.

The Burst Media study of 25+ year olds finds that over a half of women say that the World Wide Web is their main source of information when conducting research into items that they might purchase. This, naturally, has a massive impact for anyone who has females as their target customers: from car manufacturers to clothing, from technology to toys, lingere to leather sofas.

Internet useage, identified by the study, indicates that the about 70% of women are online for personal use from 7pm to midnight and about 50% online from 4-7pm shopping, browsing, researching with over 55% of females not online before 7am. During most parts of the day, over 80% people are online at any one time. These are great statistics for anyone targetting lady customers using PPC as it informs the ad scheduling that they might like to engage in.

The ‘highest percentage of people online award’ goes to the Netherlands, which has 83% of its population online, whilst Germany has the most users - at a staggering 32.5 million during may alone.

With 122 million Europeans online every day, spending on average 24.1 hours online every month, the internet has to be the main part of any marketing campaign for any serious online business.

ComScore’s Bob Ivins states that the correlation between broadband “roll-out” and internet useage is incontravertable: the more widespread broadband becomes, the greater the number of internet users (and regular internet users).

Guests included the lovely Annabel Carmel, recently returned from high profile interviews in the states, product owners, online stores and various publications.

Pricegrabber really looked after us with great cocktails and excellent canapes before the CEO thanked clients for their support over the previous year and looked towards the future synergism of Pricegrabber with its customers and to a future laden with mutual benefits and profit! The DJ then let the music hit the decks and the party rocked on…

Sound advice came from all avenues, including from CMS vendors Jadu, Motor Addicts, Headshift, Sun Microsystems and Bupa, the healthcare provider. One overall message which came from a number of speakers to the companies and associations in the audience was to be brave: relinquish control and hand over your social media to your home and outside community. Gone are the days of controlled press releases being the only avenue for information leakage, your company’s networks and community groups are the way forward in developing value, both internally and externally. Internal value is created through strengthening bonds, a greater degree of trust and responsibility, valuing contributions and individuals, organic and viral information sharing and, of course, building natural communities - especially in larger organisations. The external value of Social Media is generated through customer interations with each other and with the company, through potential clients approaching in a more informal way, whether as a ‘lurker’ (someone who reads but does not participate) or as a fully fledged community member. Sun Microsystem’s Mink Dimension, aka Simon Phillips, discussed how CEO Jonathon’s blog interaction with staff, customers and regulatory bodies(!) has created a marked increase on the company’s bottom line.

In addition to the speed networking hosted enthusiastically by LLoyd Davis of Perfect Path Consulting, the open session was a great way for the audience to mingle, network and mull over social media topics of interest. Based around subjects that were of concern for individuals, attendees offered to ‘host’ these sessions. Participants then had an ‘open foot’ policy whereby they could sit and contribute to the whole discussion or could get up and walk over to adjoining conversations.

Monetising Social Media

This worked extremely well for both those with a particular motivation - whether the dark side of Social Media, Monetizing SM (note: not S & M), Real World Social Media or Moderating posts or dealing with spam - and for those who just wanted to get the aroma of the conversations, pause for a deep inhalation or just merely whaft off to join another group (to use olfactory terminology).

Looking at Moderating Blogging & SM: pre moderation, post moderation and the implications for each method plus looking at CMS systems in tandem with blogging and social media.

The Dark Side of Social Media focused on the potenital for misuse of information. It referred to the information-gathering underworld who could employ too much data to ill intent, whether personally or professionally. Discussion also included concerns about children and their activity on the various social media forums from facebook to myspace to bebo; the recent prevention of the police by Google to use fake profiles to try to catch paedophiles and the danger that this impediment could place upon our law force; Googling potential and current employees (a phenomenon only just catching on in the UK - I think we have yet to start googling potential dates, like the US regularly do!) and how a compromising picture of a government servant / traditional organisation employee appearing could result in firing of that same individual. Also covered was how social media could be used to ‘brand’ individuals but, as someone pointed out, a facade could only be held for so long before the words belied the true persona!

All in all, a very positive day with an excellent host in the form of Euen Semple, quality attendees, great information sharing and a truly superb buffet lunch (SES take notes!) from the Marriott Grosvenor Square.

I am covering Impact of Social Media and Enterprise 2.0 conference and am very much looking forward to attending the day and, of course, meeting up with the esteemed UK SEO and Blogging community. Highlights of the event include how to manage your corporate brand through blogging, developing a social media strategy and a speed networking session where you get to chat for 3 minutes!
Watch this space….