Webrequest Not Returning Html

I want to load this http://www.yellowpages.ae/categories-by-alphabet/h.html url, but it returns null In some question I have heard about adding Cookie container but it is already t

Solution 1:

You need CookieCollection to get cookies and set UseCookie to true in HtmlWeb.

CookieCollection cookieCollection = null;
var web = newHtmlWeb
{
    //AutoDetectEncoding = true,UseCookies = true,
    CacheOnly = false,
    PreRequest = request =>
    {
        if (cookieCollection != null && cookieCollection.Count > 0)
            request.CookieContainer.Add(cookieCollection);

        returntrue;
    },
    PostResponse = (request, response) => { cookieCollection = response.Cookies; }
};

var doc = web.Load("https://www.google.com");

Solution 2:

I doubt it is a cookie issue. Looks like a gzip encryption since I got nothing but gibberish when I tried to fetch the page. If it was a cookie issue the response should return an error saying so. Anyhow. Here is my solution to your problem.

publicstaticvoidMain(string[] args)
{
    HtmlDocument doc = new HtmlAgilityPack.HtmlDocument();
    try
    {
        HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://www.yellowpages.ae/categories-by-alphabet/h.html");
        request.Method = "GET";
        request.ContentType = "text/html;charset=utf-8";
        request.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;

        using (var response = (HttpWebResponse)request.GetResponse())
        {
            using (var stream = response.GetResponseStream())
            {
                doc.Load(stream, Encoding.GetEncoding("utf-8"));
            }
        }
    }
    catch (WebException ex)
    {
        Console.WriteLine(ex.Message);
    }
    Console.WriteLine(doc.DocumentNode.InnerHtml);
    Console.ReadKey();
}

All it does is that it decrypts/extracts the gzip message that we receive. How did I know it was GZIP you ask? The response stream from the debugger said that the ContentEncoding was gzip.

Basically just add:

request.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;

To your code and you're good.