Magento get products from category, order by rand()

Magento collection do not accept parameters other then one of selected attribute. In this case you should get Zend_Db_Select object and add order instruction to it.

/* @var $products Mage_Catalog_Model_Resource_Eav_Mysql4_Product_Collection */
$products = Mage::getModel('catalog/product')
->getCollection()
->addAttributeToSort()
->addAttributeToSelect('small_image')
->addCategoryFilter(Mage::getModel('catalog/category')->load());
$products->getSelect()->order(new Zend_Db_Expr('RAND()'));
To see what query will be executed you may use this construnction

$products->load(true, true); // first parameter show sql query in output, second s