Update on the changes to brps.js

After the post about potential upcoming change to brps.js. As of writing this post, the change has been taking effective.

For last two weeks, I tried to block bad IPs, which I found some blogs use bots repeatedly to bloat their viewer statistics. And the scale is quite high, by blocking subnets, the CPU cost reducing from $3+ to $0.

However, it's still not enough, datastore operations still cause $0.03 a day at least. Which, obviously, I wouldn't want to pay for. Heck, this is free app, I won't pay for anything to service.

It's a shame, crappy blogs cause me to take some measures in order to keep this service free.

Blocking IPs is not a good idea because they come from everywhere and many legitimate requests are also blocked. So I decided to do something on the client script.

I thought I would redirect, but I would like to try to make the list loading manual first to see if that would reduce requests.

From my experiences, I know less than one percent of people would really want to read you related posts list. The actually number is lower. Some people probably not notice you have such list on your blog if your blog looks like being splashed by colorful inks.

Anyway, at this moment, you should see a button instead of a list. I have updated the documentation, you can actually use own text for the button or set up automatically loading as before. But please, if you don't have no good reason, please don't turn on autoload.

It would be better if you can switch to the newer client script.

There might be a change to brps.js

This only affects brps.js users.

Long story short: In order to keep BRPS free, I most likely would redirect brps.js script to gas.js.

Currently, brps.js uses 96% of overall processing and GAE will start to use new charging model, which is certainly cause BRPS using all free quota and more. Please read this post for little more on it.

Basically, you don't need to do anything, at least not now. But some feature might be missing, I am not quite sure if two scripts have same features. It's been too long since last time I read them.

This is just a quick update. If you see something wrong with your widget on November 1st, you know what's going on.

Future of BRPS

After I released the new client script, gas.js, I was very happy with it. Few modifications were made after, also included a piece of code for tracking usage using Google Analytics on around 9/21/2010. Here is a quick peek of statistics:

Few days ago, Google Code announced a new APIs console for easily monitoring API usage. I thought I might switch to this new stuff. But then I found out, they has deprecated Web Search API.

I'm officially doomed.

They promote Custom Search API as a successor. If you ever use Custom Search on your website, you know they are not the same. Custom Search API does provide APIs to create custom search engine and to retrieve the results. But that is not what BRPS needs. BRPS needs to search for any website which include the client script, creating custom search engine for each website isn't doable.

So, I have been thinking what options would I need to take once Web Search API is discontinued after three years:

• Accepting new brps.js client users,
• Using other search engine API, such as Yahoo!,
• Implementing Version 2 of BRPS,
• Waiting for enough developers whimpering and Google make changes to Custom Search API, or, my best option,
• Waiting for Blogger to release Related Posts gadget.

Who knows, we may see a new post on Blogger Buzz saying Related Posts is ready for you to try out.

So, the future of BRPS is we are still waiting to see.

New client script!

New client script! Faster and No Key required and No Bans!

No needs to ask for unblocking, no need to get the key. You just install new code, only three lines, too!

Here is quick list:

• It's faster. Because it do not use BRPS database, therefore BRPS do not need to search for labels via Blogger API. It directly uses Google Search with labels as search keyword.
• It supports multi-blog, even multi-site, because it uses Google search operator site:example.com | site:someblog.blogspot.com | ...
• It can now work on any page of your Blogger blog, but it will use all labels from that page to do the search. In other words, the search keywords can come from labels of different posts on that page which user is visiting. It will only use top 20 frequent label to query.
• No Key required! No blocking!

Go get it!

Installing BRPS in classic templates

Someone asked how to install in classic templates, so I checked up and found the code provided from BRPS wasn't working. There is an issue with HTML, so I updated the code. It should be working in classic templates.

The question is where to install those three lines. My suggestion is

At sidebar

Find the following and insert the code before:

</div></div>
<!-- End #sidebar -->

The related list will show up at the bottom of sidebar.

If you want to move the list up, just observe the code before these two lines, you can see each gadget of sidebar is a block of code, put the BRPS code between, usually above <h2> will do.

At post end

Find the following and insert the code before:

  </div>
  <!-- End .post -->

Hope this helps.

Planning Version 2

Firstly to clarify, nothing is getting started, don't be excited.

Here is some ideas:

• Label Exclusion [group]
• List Order: alphabetical, oldest-to-newest, newest-to-oldest, etc. [blog]
• Preview thumbnail
• Score Indicator (bar)
• Score Threshold: Absolute value, percentage
• Layouts: Plain text list, vertical, horizontal
• Authenticaion
• Open ID authentication: No more key (or just OAuth?)
• Support private blogs: AuthSub or OAuth
• Label Weighting and Content Weighting
• View Count
• Click Count
• View and Click Statistics
• Popular Posts based on Views and Clicks
• Non-single Post Page Allowed

I am asking you for help if you would like to see these above. If you know Python, JavaScript, or webdesign and you are willing to spend time to develop version 2, leave me a message.

I have no schedule to see those happening, so please don't expect to see version 2. I might or might not to code for it if no one joins. Unless someone is with me doing development, it's most likely, version 2 wouldn't be born.

If I start, it will be developing in another GAE app and accept few testers. Once it's done, that app will be deleted and move back to brps since brps is a short app id. The code is still Open Source and the service will still free and you still won't see any ads on your blogs or links to here if  client script runs successfully.

The design process will be open for discussion somewhere (the group, I think) and the code will be still at Google Code same repo.

I guess that's all, now it's your move.

Visualization of BRPS Code

This post is off-topic but I feel it's fun to share. The following videos are generated by Gource. They visualize how the code of BRPS were committed.

BRPS had stayed in one of my repos, the first one is a extraction version of that repo. As of April 2009, it moved into its only repo, and the second video shows the activities in its repo after.

Go to video page on YouTube

Go to video page on YouTube

If this kind of video catch your eyes, here are more of mine.

[Apology] Blocking by mistake

As of 1/6/2010, I modified the Blocking Policy, I added

"Failed installation/script removed blogs — Script installed without key or incomplete installation. Blogs have removed the script. (Added on 1/6/2010)"

At beginning of this year, many blogs have reach 3-month reviewing period. So, I started to review those blogs which haven't got accepted or blocked. I found out many of them are not using Key or installed the script incorrectly causing BRPS not working. I don't bother to notify the blogs owners, I don't have that free time to do so. Besides, the error message has been showing up for more than a month, how could a blogger not checking there own posts after publishing?

The second part of the addition, "Blogs have removed the script," it's also true that many blogs have removed the script after installing. I see no harm to block them since they don't use BRPS anymore, by blocking them so they won't show up after 3 month for reviewing.

But I made a huge mistake, because I check HTML source code in home page. If you install the script into your Template, not by adding a new HTML/JavaScript gadget, then the script wouldn't show up, they would only show up in a single post page.

I forgot that part, therefore I must have blocked some. By the time I found out, I had no idea how many had been blocked by that mistake.

So, if you install the code into your Template, just go ahead to ask for unblocking. I will check up as soon as possible. (I will still review first to check your content)

Sincere apology to you.
Sorry, again!

Ads On

I have put ads on website and this blog. Don't worry, your related list will never ever be contaminated.

Need your idea!

I believe I can claim that BRPS is a relative stable app now. So I think I have to move on, find my next target to work on.

So, here, I am asking for your idea. It can be anything to help you blog, another widget perhaps? Or you can jump out of this Blogger thing, anything could help you on anything as long as it is programmable. Hey, I do coding, remember?

Please do not be shy to respond with this! Shout out your ideas, I will do the rest for you!

And it will be free and open source!

The first donation to BRPS

Be honest, I didn't expect to receive any donation at the time I put on the donation button. I silently put it on website and blog without writing about it in any place. About 8 hours ago, I received the first donation to BRPS.

I am very thankful for the support from this donor, Deborah Swain, go check out her blog, Living in Rome.

If you also want to donate, please go to this newly created donation page and please read the notice before you donating.

Thank you for reading this!

PS. this would be the only post about donation and donor. I posted it because it's the first.

About Loading...

I think I must address this issue because of newly added change. When you see "Loading..." and it takes a lifetime to stay in your way. It does not mean anything but the server encounter the problem, it simply replies 500 Internal Server Error to client-side JavaScript (brps.js).

In plain English, there might be a bug in BRPS and it couldn't resolve it, it couldn't fail gracefully, so you never get a nice error message.

The client-side JS does have a code to deal with it, to catch the error, but I do not know why it does not work. I will be investigating it when I want to .

Again, "Loading..." do not mean your blog is blocked, is accepted, is unaccepted, it simply says BRPS goes wrong with something.

Why Enforce Using The Key?

(Last updated on 2009-11-27)

The reasons are simple,

1. I want to prevent some people keep duplicate blogs with same template.
2. I want to prevent pre-installed template, not every blog owner wants Related Posts.
3. I want to make sure you have the chance to read the Blocking Policy.
   

The new key is easier to install, you only put ?key=[YOURKEY] into the previous code. You can get the key and the instruction here.

One key can only be used for one specific blog and this is the point to stop those badass bloggers (They don't deserve that title). They have to manually get the key for each blog. If you are a normal blog, this will not take you much effort because you should only have one or a few blogs.

This change will only affect unaccepted blogs. If you do not see a message to ask you to add this new key, then it means your blog is accepted.

If you have more questions, please post at discussion group.

Updated on 2009-11-27 11:15 pm UTC

I made few change to the code and without checking. So you may have see "Loading..." and no more on your blog. It should be resolved and "Loading..." has nothing to do with being blocked or being (un)accepted. It's just a code getting stuck, it's an error on server side.

And you have not updated your installation code and saw "Could not find the key...," please use your browser refresh button to make sure you get the latest client-side JavaScript. If your blog is accepted, it should load the related list without the key specified, or you will be asked to go to obtain a key.

The problem should be gone as you read this update, sorry for the inconvenience.

And this is for record, the completed requests:

I have no idea why 8pm UTC could still have those request completed. Anyway BRPS would be having very light server load for a long time. :-)

Yeah! BRPS is one year old!

One year ago (11/11/2008), I announced that BRPS had been bringing online. It reached 1000 requests/day in less a month. A month old, it ran of free quota because more than 200,000 requests a day. Therefore, I started to tweak the code to make it run faster and use less resources. I also implemented a reviewing system to allow me easily to block some bad blogs.

The results are great. I doubled the capacity with same amount of resource. Currently it has 160,000+ requests a day (1.85+ completed requests/second) and 350+ active blogs every hour. More than 2,000 blogs and 50,000 posts are stored in datastore.

Currently, it has 2177 blogs registered, only 140 blogs are accepted (6.43 %) and 831 blogs are blocked (38.17 %). You may think blocking rate is quite high. In deed, there are a lot of bad blogs on Blogger. I have to honey to say after I started to review blogs, I have tagged people from certain country who using Blogger (Only those use blogs, they still have good and talent people, a lot!) with very low value. 95% of their blogs are involving copyright infringement. All posts were copied from here and there, they don't even have rights to claim fair use. Some of them are suspicious to using automatically method to create mass blogs and posts. They are just awful and without moral and honor.

Statistics Page

I had thought about to expand to BRPS' features, but the resources didn't allow me to do that. I want to keep free. The incoming bandwidth (750 MB out of 1000 MB) is the current one would probably to exceed the quota. If BRPS have more bloggers use it, soon, it will blow and I have no way to avoid it.

Dashboard of BRPS in GAE

Quota Details

Anyway, Happy Birthday to BRPS!

Blogger down, BRPS knows

Hours ago, I tried to get into Blogger dashboard but I got bX-kn2493
 then I thought it would be fixed soon. After a while, I went to do reviewing, then I found out requests count had a big drop. I checked one of my blogs, I got bX-59cppw, I realized it's whole Blogger down.

Here are two charts, the first one was taken from App Engine, it shows Requests/Second versus Last 24-hour time:

Second one is the Completed Requests over Last 24-hour time:

Last one is the Active Blogs over Last 24-hour time:

I wonder what Blogger did, maybe it's just a hardware failure?

About Reviewing

I review every blog, usually it only takes me five to thirty second to decide if this blog should be marked as

• Reviewed — I think it's okay and BRPS will notice me this blog again 90 days after this reviewing.
• Accepted — I gives it green light, it's a good blog, I don't need to review it anymore.
• Blocked — It violates Blocking Policy.

Here is a quick screenshot of reviewing interface:

Generally, if your blog is quite unique—you customize your layout quite good and don't have crap Flash or images, then your blog should be marked as accepted. If you use those awful-professional template, it could be marked as reviewed.

Note that you could only know if your blog is blocked or not, you wouldn't know if it's reviewed or accepted. If your blog is blocked, you will get a warning message instead of a list of related posts.

Blocking Policy

(Last updated on 1/6/2010)

BRPS is a free service, it uses free quota of what Google App Engine provided. It wasn't planned to charge, however the quota is limited and lots of blog using BRPS, therefore some blogs may not be using BRPS.

Any blog is having the following

• Copyright infringement (If your blogs have links to any illegal downloads, directly or indirectly, upload by youself or not, please DO NOT use BRPS. Otherwise, your blogs will be BLOCKED for sure),
• Hate against a protected group,
• Adult or pornographic images,
• Promotion of dangerous and illegal activity,
• Content facilitating phishing or account hijacking,
• Impersonated user identity, or
• Abuse this service.

will definitely be blocked.

There are more could be used as blocking reasons:

• Similar contents — There are few blogger (useless content creator) creating lots of blogs with very similar content in each of their blogs. Those contents are useless, the primary purpose is to expose their blogs and get revenues.
• Referral contents ­— Almost all posts have link to Amazon or some online stores with their referral code. Those blogs are just created to make money.
• Few posts — BRPS automatically block blogs only have less than 20 posts. There is no much benefit of using BRPS. Please don't waste server resources. If you got blocked, request a unblocking review once your blog post count reaches 20. (Added on 10/18/2009, updated on 11/17/2009)
  
• Test blog—If you are testing BRPS on your sandbox blog, that blog will be blocked. (Added on 10/18/2009)
• Failed installation/script removed blogs — Script installed without key or incomplete installation. Blogs have removed the script. (Added on 1/6/2010)

The reasons above are not the only ones, your blogs could be blocked for any reason.

If your blogs are blocked and you believe your blogs shouldn't be deserved for that, please report with your blog URL.

You can still host your own BRPS on Google App Engine, you can grab the code and run on your own. Please DO NOT ask for help about self hosting!

Special note to template developer: Please DO NOT activate BRPS if you add it into your template creations. Comment out the script! And write a instruction to guide your user to activate it.Not everyone needs Related Posts in their blog. (Added on 10/18/2009) (Removed on 1/6/2010)

Explanation of “No related posts found.”

The blogger of As the crow flies (this blog has good photos, go check it out) asked a question about why is one of her post gets no results. I have seen many blog posts seem to have same problem from server logs, but this is the first time I got asked. Probably a good time to address the issue for you.

Here is a short answer:

When BRPS inquires about

• same label posts, or
  
• specific post's labels

to Google Blogger, sometimes, Blogger replies with 404 NOT FOUND. Which might mean there is

• no posts under such label, or
  
• no such post,

but there is actually some posts are filed under such label or there indeed has that post. The cause is unknown to me, and I am not able to resolve this. When this happens, it may result “No related posts found.” to your blog posts.

Fortunately, the problem is not permanent. If BRPS inquires again later, it may get the correct result if Blogger no long respond 404. But, since BRPS has 24 hours cache time for each post's labels and same label posts list, the result will have to wait at least 24 hours (up to 48 hours) to get updated.

Please bear with this issue if this badly happens on your blogger.

Moved here

All new blog posts will be posted here, you can read old posts here.